00000100:02000000:0.0F:1521773263.363016:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0F:1521773263.363190:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.363193:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.363196:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2671f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.363199:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0F:1521773263.363236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.363248:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.376564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.376758:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.376760:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.376763:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26721 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.376766:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.376803:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.376813:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.389527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.389674:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.389676:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.389679:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26723 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.389682:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.389703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.389714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.399488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.399698:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.399700:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.399703:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26725 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.399706:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.399728:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.399739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.409372:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.409516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.409518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.409521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26727 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.409523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.409546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.409556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.418847:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.418993:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.418995:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.418998:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26729 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.419000:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.419022:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.419033:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.432451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.432622:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.432624:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.432627:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2672b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.432629:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.432658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.432669:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.441697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.441856:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.441858:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.441861:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2672d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.441864:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.441886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.441897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.455470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.455631:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.455635:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.455638:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2672f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.455640:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.455666:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.455677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.468886:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.469104:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.469107:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.469109:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26731 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.469112:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.469133:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.469143:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.478571:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.478710:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.478712:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.478715:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26733 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.478717:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.478740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.478750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.492452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.492608:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.492610:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.492612:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26735 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.492615:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.492637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.492648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.502394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.502550:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.502552:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.502555:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26737 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.502557:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.502582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.502593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.512050:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.512191:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.512193:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.512196:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26739 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.512198:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.512219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.512235:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.521940:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.522078:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.522080:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.522082:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2673b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.522085:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.522107:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.522118:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.535720:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.535900:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.535902:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.535905:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2673d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.535907:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.535930:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.535940:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.548934:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.549076:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.549080:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.549083:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2673f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.549085:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.549108:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.549118:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773263.562565:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.562723:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.562726:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.562728:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26741 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.562731:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.562753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.562764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.588193:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.588391:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.588393:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.588396:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26743 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.588399:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.588419:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.588430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.602192:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.602344:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.602359:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.602362:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26745 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.602365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.602392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.602403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.615597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.615710:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.615711:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.615713:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26747 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.615716:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.615742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.615754:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.628753:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.628851:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.628853:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.628856:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26749 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.628858:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:2.0:1521773263.628878:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:2.0:1521773263.628889:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.642801:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.642944:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.642946:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.642949:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2674b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.642952:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.642965:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.642976:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.656389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.656811:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.656812:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.656815:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2674d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.656817:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.656829:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.656854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.670567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.670666:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.670668:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.670671:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2674f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.670673:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.670688:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.670698:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.685241:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.685356:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.685358:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.685361:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26751 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.685380:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.685395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.685420:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.699552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.699654:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.699656:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.699658:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26753 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.699661:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.699675:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.699685:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.713445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.713548:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.713549:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.713552:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26755 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.713554:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.713569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.713578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.722804:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.722893:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.722894:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.722897:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26757 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.722899:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.722931:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.722940:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.736438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.736561:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.736563:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.736567:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26759 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.736569:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.736589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.736600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.745855:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.745973:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.745974:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.745977:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2675b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.745979:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.745993:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.746002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.755793:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.755902:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.755904:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.755906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2675d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.755908:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.755922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.755932:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.765504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.765614:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.765616:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.765618:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2675f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.765621:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.765636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.765646:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.775426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.775524:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.775526:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.775528:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26761 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.775531:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.775544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.775553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.785162:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.785546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.785548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.785550:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26763 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.785553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.785568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.785584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.798774:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.798903:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.798904:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.798907:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26765 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.798909:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.798922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.798933:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.808389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.808488:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.808490:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.808492:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26767 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.808494:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.808512:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.808521:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.821438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.821554:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.821556:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.821559:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26769 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.821561:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.821578:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.821594:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.830563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.830666:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.830668:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.830670:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2676b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.830673:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.830687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.830697:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.843546:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.843649:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.843651:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.843654:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2676d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.843656:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.843671:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.843680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.856991:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.857094:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.857095:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.857098:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2676f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.857100:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.857114:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.857128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.865566:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.865675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.865677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.865681:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26771 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.865683:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.865700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.865711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.874781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.874888:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.874890:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.874892:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26773 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.874895:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.874922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.874949:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.884155:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.884254:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.884255:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.884258:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26775 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.884260:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.884305:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.884337:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.897649:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.897754:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.897755:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.897758:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26777 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.897760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.897775:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.897800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.907888:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.907994:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.907996:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.907999:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26779 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.908002:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.908035:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.908045:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.918313:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.918418:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.918420:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.918422:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2677b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.918425:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.918437:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.918451:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.932416:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.932519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.932521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.932523:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2677d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.932526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.932538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.932548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.946563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.946670:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.946672:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.946674:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2677f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.946677:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.946690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.946700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.960593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.960695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.960697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.960700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26781 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.960702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.960714:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.960730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.975139:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.975252:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.975254:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.975271:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26783 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.975273:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.975285:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.975311:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773263.988770:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773263.988870:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773263.988871:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773263.988874:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26785 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773263.988876:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773263.988888:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773263/real 1521773263]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773275 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773263.988914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773264.003585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.003686:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.003688:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.003690:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26787 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.003693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.003707:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.003723:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773264.024163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.024317:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.024318:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.024321:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26789 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.024323:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.024335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.024394:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773264.038409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.038517:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.038519:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.038521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2678b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.038523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.038536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.038546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773264.052909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.053016:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.053018:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.053020:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2678d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.053023:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.053034:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.053067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.067362:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.067468:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.067470:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.067473:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2678f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.067476:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.067488:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.067498:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.080683:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.080791:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.080794:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.080796:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26791 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.080799:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.080812:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.080822:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.093964:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.094081:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.094084:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.094086:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26793 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.094089:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.094101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.094120:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.107446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.107567:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.107569:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.107573:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26795 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.107575:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.107587:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.107595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.118859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.118973:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.118975:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.118978:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26797 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.118980:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.118993:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.119002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.132419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.132549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.132551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.132554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26799 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.132557:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.132570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.132580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.143950:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.144056:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.144058:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.144061:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2679b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.144063:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.144073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.144083:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.153554:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.153662:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.153664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.153667:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2679d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.153670:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.153682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.153692:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.166983:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.167091:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.167093:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.167096:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2679f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.167099:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.167111:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.167120:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.180437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.180566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.180568:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.180571:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267a1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.180574:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.180587:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.180597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.193659:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.193789:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.193791:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.193794:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267a3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.193796:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.193809:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.193820:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.203560:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.203672:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.203674:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.203677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267a5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.203679:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.203692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.203702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.221340:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.221461:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.221463:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.221465:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267a7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.221468:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.221479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.221489:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.234571:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.234690:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.234692:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.234695:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267a9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.234697:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.234710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.234720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.244167:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.244290:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.244292:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.244296:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267ab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.244299:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.244311:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.244321:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.254310:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.254431:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.254433:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.254436:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267ad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.254438:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.254450:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.254459:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.267963:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.268090:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.268092:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.268094:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267af sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.268097:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.268109:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.268119:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.281336:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.281476:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.281478:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.281481:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267b1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.281483:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.281496:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.281505:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.290945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.291052:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.291054:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.291057:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267b3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.291060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.291070:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.291079:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.300994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.301111:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.301113:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.301115:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267b5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.301118:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.301129:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.301138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.310395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.310524:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.310526:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.310528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267b7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.310531:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.310547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.310557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.323607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.323724:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.323726:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.323729:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267b9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.323731:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.323742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.323753:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.337075:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.337204:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.337206:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.337208:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267bb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.337210:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.337223:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.337232:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.350559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.350672:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.350675:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.350678:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267bd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.350680:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.350693:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.350703:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.363925:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.364042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.364044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.364047:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267bf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.364049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.364062:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.364071:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.377685:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.377798:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.377800:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.377803:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267c1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.377806:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.377818:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.377826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.387592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.387708:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.387710:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.387713:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267c3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.387715:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.387728:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.387738:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.401079:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.401208:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.401210:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.401213:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267c5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.401215:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.401228:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.401237:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.411228:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.411369:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.411371:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.411393:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267c7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.411396:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.411409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.411435:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.425476:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.425591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.425593:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.425595:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267c9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.425598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.425614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.425624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.440060:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.440173:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.440175:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.440177:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267cb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.440180:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.440192:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.440218:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.454305:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.454418:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.454420:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.454423:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267cd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.454425:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.454438:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.454448:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.467924:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.468054:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.468056:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.468059:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267cf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.468061:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.468073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.468101:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.481368:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.481479:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.481481:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.481484:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267d1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.481486:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.481498:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.481536:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.495733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.495859:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.495861:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.495864:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267d3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.495866:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.495878:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.495898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.510620:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.510743:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.510745:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.510748:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267d5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.510751:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.510763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.510773:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.530152:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.530261:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.530263:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.530267:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267d7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.530270:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.530283:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.530309:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.543549:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.543664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.543666:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.543669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267d9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.543671:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.543685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.543711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.557536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.557669:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.557671:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.557674:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267db sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.557676:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.557691:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.557701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.572006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.572203:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.572219:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.572222:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267dd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.572224:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.572236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.572257:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.585871:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.586013:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.586015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.586018:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267df sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.586020:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.586032:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.586058:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.600169:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.600282:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.600284:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.600287:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267e1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.600289:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.600301:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.600328:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.614578:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.614690:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.614693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.614695:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267e3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.614698:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.614711:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.614720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.628690:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.628814:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.628816:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.628819:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267e5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.628821:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.628833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.628844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.642574:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.642689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.642691:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.642694:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267e7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.642696:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.642709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.642719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.653403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.653562:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.653564:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.653567:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267e9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.653570:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.653590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.653602:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.663261:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.663389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.663391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.663394:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267eb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.663397:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.663409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.663419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.673543:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.673660:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.673662:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.673666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267ed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.673669:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.673679:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.673689:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.683334:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.683455:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.683457:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.683460:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267ef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.683462:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.683474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.683484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.693175:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.693286:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.693288:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.693290:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.693293:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.693305:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.693314:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.702905:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.703025:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.703027:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.703030:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.703033:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.703043:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.703052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.713052:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.713161:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.713163:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.713166:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.713168:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.713181:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.713190:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.723061:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.723167:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.723169:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.723172:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.723174:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.723185:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.723195:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.737081:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.737191:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.737193:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.737196:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.737198:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.737210:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.737219:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.750800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.750931:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.750933:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.750936:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.750939:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.750951:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.750961:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.761074:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.761190:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.761192:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.761195:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.761197:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.761210:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.761220:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.774589:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.774709:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.774711:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.774714:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x267ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.774716:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.774729:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.774739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.784079:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.784202:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.784204:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.784206:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26801 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.784209:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.784221:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.784230:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.797574:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.797727:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.797729:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.797733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26803 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.797736:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.797746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.797756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.807441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.807545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.807547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.807550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26805 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.807552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.807563:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.807573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.817200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.817322:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.817324:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.817327:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26807 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.817329:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.817341:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.817376:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.831114:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.831243:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.831245:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.831248:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26809 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.831250:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.831263:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.831272:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.839899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.840027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.840029:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.840032:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2680b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.840034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.840046:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.840055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.849753:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.849870:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.849872:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.849875:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2680d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.849878:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.849890:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.849899:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.863201:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.863328:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.863330:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.863332:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2680f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.863335:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.863373:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.863383:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.876601:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.876734:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.876736:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.876739:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26811 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.876741:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.876754:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.876764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.886032:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.886164:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.886166:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.886169:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26813 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.886171:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.886183:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.886193:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.899572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.899697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.899699:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.899702:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26815 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.899704:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.899717:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.899726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.913220:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.913358:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.913360:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.913363:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26817 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.913365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.913375:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.913384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.923236:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.923358:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.923359:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.923363:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26819 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.923365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.923375:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.923383:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.936474:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.936592:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.936594:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.936597:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2681b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.936600:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.936613:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.936623:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.950067:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.950284:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.950286:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.950289:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2681d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.950291:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.950304:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.950313:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.959567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.959682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.959684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.959687:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2681f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.959689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.959702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.959711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.969271:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.969402:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.969404:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.969407:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26821 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.969409:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.969421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.969430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.983069:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.983255:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.983257:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.983259:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26823 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.983262:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.983282:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.983292:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773264.996577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773264.996722:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773264.996724:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773264.996727:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26825 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773264.996729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773264.996741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773264/real 1521773264]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773276 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773264.996751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.010413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.010532:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.010534:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.010537:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26827 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.010539:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.010556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.010567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.020443:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.020668:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.020670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.020673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26829 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.020676:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.020689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.020699:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.047920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.048050:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.048052:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.048055:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2682b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.048057:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.048069:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.048079:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.061607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.061767:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.061769:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.061785:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2682d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.061787:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.061800:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.061809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.072494:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.072709:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.072711:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.072715:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2682f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.072717:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.072730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.072739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.104943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.105110:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.105114:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.105120:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26831 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.105125:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.105149:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.105169:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.115842:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.115959:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.115961:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.115964:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26833 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.115966:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.115979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.115988:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.125452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.125586:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.125588:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.125590:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26835 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.125592:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.125605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.125614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.138978:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.139095:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.139097:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.139100:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26837 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.139102:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.139114:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.139123:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.183465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.183593:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.183595:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.183598:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26839 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.183600:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.183618:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.183628:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.191145:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.191258:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.191260:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.191263:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2683b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.191265:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.191278:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.191287:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.204906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.205022:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.205024:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.205027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2683d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.205030:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.205039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.205050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.218406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.218528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.218530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.218533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2683f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.218535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.218548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.218558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.228691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.228808:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.228810:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.228813:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26841 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.228816:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.228828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.228838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.279596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.279741:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.279743:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.279746:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26843 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.279748:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.279760:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.279769:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.293319:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.293460:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.293462:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.293466:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26845 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.293468:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.293480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.293490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.307181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.307296:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.307298:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.307301:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26847 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.307303:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.307331:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.307340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.317557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.317679:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.317681:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.317684:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26849 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.317687:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.317697:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.317707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.328012:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.328127:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.328129:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.328132:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2684b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.328135:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.328147:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.328156:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.375420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.375590:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.375592:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.375595:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2684d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.375597:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.375617:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.375628:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.389256:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.389397:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.389399:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.389402:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2684f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.389404:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.389414:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.389424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.402839:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.402958:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.402960:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.402962:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26851 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.402965:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.402978:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.402987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.416380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.416536:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.416539:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.416541:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26853 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.416544:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.416556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.416567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.425915:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.426025:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.426027:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.426029:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26855 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.426032:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.426044:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.426053:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.470971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.471099:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.471101:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.471104:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26857 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.471106:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.471118:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.471127:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.480947:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.481061:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.481063:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.481066:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26859 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.481068:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.481081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.481090:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.491447:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.491566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.491569:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.491573:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2685b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.491575:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.491588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.491598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.501124:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.501234:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.501236:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.501239:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2685d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.501241:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.501252:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.501260:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.514569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.514685:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.514687:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.514690:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2685f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.514692:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.514720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.514730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.524837:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.524961:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.524963:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.524966:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26861 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.524968:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.524980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.524990:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.557118:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.557251:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.557253:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.557256:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26863 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.557258:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.557271:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.557280:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.582150:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.582265:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.582267:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.582270:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26865 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.582272:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.582284:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.582293:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.591917:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.592043:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.592045:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.592048:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26867 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.592050:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.592063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.592072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.605645:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.605784:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.605786:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.605789:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26869 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.605791:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.605805:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.605815:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.619174:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.619316:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.619318:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.619321:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2686b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.619323:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.619335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.619344:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.628623:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.628745:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.628747:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.628750:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2686d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.628752:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.628765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.628774:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.638413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.638549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.638551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.638554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2686f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.638557:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.638570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.638579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.648302:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.648434:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.648436:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.648441:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26871 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.648443:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.648454:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.648463:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.658171:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.658284:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.658285:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.658288:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26873 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.658291:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.658303:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.658312:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.668189:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.668310:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.668312:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.668314:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26875 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.668317:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.668354:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.668381:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.682145:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.682266:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.682268:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.682271:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26877 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.682273:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.682285:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.682295:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.691780:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.691892:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.691894:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.691897:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26879 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.691899:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.691911:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.691920:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.701610:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.701729:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.701731:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.701734:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2687b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.701737:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.701750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.701760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.710965:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.711074:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.711076:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.711078:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2687d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.711081:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.711091:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.711100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.721502:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.721621:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.721623:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.721626:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2687f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.721628:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.721638:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.721648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.735308:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.735432:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.735434:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.735437:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26881 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.735439:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.735452:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.735461:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.748795:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.748923:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.748925:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.748928:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26883 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.748930:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.748943:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.748952:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.762326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.762453:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.762455:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.762458:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26885 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.762460:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.762473:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.762483:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.773088:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.773208:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.773210:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.773214:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26887 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.773216:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.773228:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.773237:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.786876:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.787004:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.787006:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.787009:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26889 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.787011:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.787024:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.787033:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.796301:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.796411:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.796428:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.796431:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2688b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.796433:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.796444:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.796454:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.806503:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.806632:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.806634:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.806637:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2688d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.806640:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.806650:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.806660:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.820429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.820542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.820544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.820547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2688f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.820549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.820562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.820571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.830415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.830537:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.830539:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.830542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26891 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.830544:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.830561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.830571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.839737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.839865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.839867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.839870:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26893 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.839872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.839884:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.839893:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.850031:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.850146:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.850149:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.850151:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26895 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.850154:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.850179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.850189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.859923:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.860042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.860044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.860047:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26897 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.860049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.860061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.860070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.873584:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.873699:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.873701:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.873704:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26899 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.873707:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.873717:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.873727:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.886951:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.887086:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.887088:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.887091:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2689b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.887093:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.887106:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.887115:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.900573:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.900684:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.900686:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.900691:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2689d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.900693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.900705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.900715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.910415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.910533:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.910535:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.910538:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2689f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.910540:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.910557:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.910567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.924089:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.924225:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.924227:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.924230:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268a1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.924232:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.924243:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.924252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.937608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.937742:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.937744:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.937747:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268a3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.937749:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.937761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.937786:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.964078:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.964214:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.964216:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.964219:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268a5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.964221:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.964235:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.964244:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.973481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.973660:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.973662:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.973664:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268a7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.973666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.973680:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.973689:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773265.987413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773265.987568:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773265.987570:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773265.987573:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268a9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773265.987575:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773265.987594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773265/real 1521773265]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773277 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773265.987615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.000707:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.000855:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.000857:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.000859:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268ab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.000862:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.000874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.000884:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.010006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.010139:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.010156:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.010159:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268ad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.010161:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.010172:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.010181:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.019592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.019734:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.019736:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.019739:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268af sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.019741:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.019752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.019760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.032755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.032879:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.032881:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.032884:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268b1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.032886:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.032899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.032908:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.046018:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.046144:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.046146:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.046150:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268b3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.046152:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.046165:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.046174:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.055416:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.055565:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.055567:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.055570:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268b5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.055572:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.055581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.055590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.068539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.068669:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.068671:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.068674:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268b7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.068676:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.068689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.068699:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.078566:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.078688:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.078690:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.078693:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268b9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.078695:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.078707:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.078716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.091932:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.092043:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.092045:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.092047:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268bb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.092050:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.092063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.092072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.101539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.101663:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.101665:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.101668:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268bd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.101670:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.101683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.101693:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.111821:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.111933:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.111935:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.111938:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268bf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.111940:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.111953:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.111963:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.121204:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.121332:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.121334:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.121336:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268c1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.121339:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.121360:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.121371:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.130863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.130992:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.130994:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.130996:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268c3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.130998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.131010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.131036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.140611:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.140718:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.140720:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.140723:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268c5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.140725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.140738:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.140748:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.150891:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.151018:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.151020:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.151023:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268c7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.151025:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.151037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.151047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.160488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.160613:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.160615:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.160619:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268c9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.160621:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.160633:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.160642:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.170337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.170467:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.170469:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.170472:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268cb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.170474:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.170487:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.170496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.180651:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.180791:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.180793:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.180796:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268cd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.180799:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.180812:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.180821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.190828:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.190953:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.190955:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.190958:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268cf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.190960:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.190970:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.190980:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.204561:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.204673:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.204675:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.204678:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268d1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.204680:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.204693:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.204702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.214217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.214373:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.214375:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.214378:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268d3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.214381:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.214391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.214401:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.227736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.227867:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.227869:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.227872:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268d5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.227874:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.227887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.227897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.237395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.237498:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.237500:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.237503:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268d7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.237505:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773266.237523:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773266.237534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.247486:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773266.247608:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773266.247610:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:2.0:1521773266.247612:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268d9 sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773266.247615:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.247630:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.247640:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.257472:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.257603:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.257605:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.257608:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268db sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.257610:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.257629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.257645:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.267427:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.267538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.267540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.267543:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268dd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.267545:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.267569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.267579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.277342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.277496:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.277498:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.277501:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268df sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.277503:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.277543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.277553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.287115:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.287210:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.287212:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.287215:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268e1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.287217:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.287229:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.287239:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.296765:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.296887:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.296891:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.296902:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268e3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.296907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.296933:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.296943:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.310324:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.310446:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.310448:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.310461:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268e5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.310463:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.310489:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.310504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.320382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.320492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.320494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.320496:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268e7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.320499:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.320540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.320550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.330562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.330665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.330667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.330670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268e9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.330672:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.330689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.330699:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.340245:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.340381:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.340386:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.340400:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268eb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.340405:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.340423:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.340433:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.350342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.350479:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.350481:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.350484:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268ed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.350486:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.350503:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.350538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.360734:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.360858:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.360860:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.360863:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268ef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.360865:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.360882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.360896:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.370721:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.370846:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.370848:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.370850:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.370853:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.370868:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.370878:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.380571:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.380673:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.380675:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.380677:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.380680:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.380696:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.380705:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.394171:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.394279:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.394281:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.394284:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.394286:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.394306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.394316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.404235:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.404361:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.404363:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.404365:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.404368:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.404386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.404395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.428187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.428493:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.428495:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.428498:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.428500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.428547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.428565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.448925:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.449028:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.449030:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.449033:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.449035:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.449053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.449062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.463380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.463499:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.463501:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.463503:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.463506:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.463547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.463557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.477433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.477536:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.477538:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.477540:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x268ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.477543:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.477560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.477570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.490971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.491079:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.491081:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.491084:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26901 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.491086:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.491101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.491110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.501132:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.501204:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.501208:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26903 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773266.501210:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.501220:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.501249:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.516709:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.516815:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.516817:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.516820:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26905 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.516822:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.516843:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.516853:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.527383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.527487:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.527489:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.527491:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26907 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.527494:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.527517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.527543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.536832:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.536942:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.536946:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.536948:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26909 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.536950:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.536965:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.536975:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.551151:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.551243:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.551246:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2690b sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773266.551249:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.551268:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.551278:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.565250:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.565358:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.565360:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.565363:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2690d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.565365:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.565383:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.565400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.575099:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.575216:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.575218:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.575221:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2690f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.575223:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.575238:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.575247:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.588697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.588805:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.588807:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.588810:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26911 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.588812:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.588849:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.588858:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.598619:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.598717:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.598719:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.598722:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26913 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.598725:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.598742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.598752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.612419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.612494:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.612498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26915 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773266.612501:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.612540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.612550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.622174:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.622276:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.622278:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.622280:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26917 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.622283:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.622298:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.622313:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.636024:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.636127:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.636129:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.636131:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26919 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.636135:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.636154:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.636163:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.646469:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.646566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.646568:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.646571:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2691b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.646573:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.646585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.646595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.657133:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.657226:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.657228:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.657231:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2691d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.657233:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.657248:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.657257:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.667629:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.667735:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.667737:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.667740:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2691f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.667742:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.667759:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.667769:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.678039:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.678158:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.678160:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.678163:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26921 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.678165:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.678180:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.678195:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.687848:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.687958:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.687960:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.687963:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26923 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.687965:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.687980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.687989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.701795:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.701914:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.701916:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.701918:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26925 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.701921:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.701941:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.701950:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.711873:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.711977:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.711979:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.711998:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26927 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.712000:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.712010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.712019:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.722434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.722529:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.722531:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.722533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26929 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.722537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.722548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.722558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.732087:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.732203:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.732204:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.732207:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2692b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.732209:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.732230:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.732246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.741721:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.741845:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.741847:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.741850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2692d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.741852:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.741868:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.741877:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.751652:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.751756:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.751758:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.751761:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2692f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.751764:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.751781:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.751791:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.761553:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.761656:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.761658:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.761661:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26931 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.761663:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.761674:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.761684:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.771136:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.771250:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.771252:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.771255:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26933 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.771257:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.771272:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.771281:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.780616:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.780719:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.780721:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.780723:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26935 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.780726:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.780741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.780757:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.790284:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.790396:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.790398:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.790400:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26937 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.790402:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.790433:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.790442:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.803935:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.804047:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.804049:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.804051:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26939 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.804055:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.804074:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.804084:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.818053:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.818159:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.818161:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.818164:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2693b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.818166:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.818188:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.818197:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.831860:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.831979:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.831981:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.831984:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2693d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.831986:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.832006:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.832016:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.841891:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.842231:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.842233:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.842235:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2693f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.842238:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.842252:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.842267:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.856090:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.856221:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.856223:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.856226:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26941 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.856228:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.856247:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.856256:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.865432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.865537:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.865539:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.865542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26943 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.865544:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.865556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.865565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.878740:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.878848:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.878850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.878853:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26945 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.878855:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.878876:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.878885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.888340:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.888467:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.888469:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.888471:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26947 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.888473:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.888488:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.888498:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.897939:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.898297:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.898299:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.898302:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26949 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.898306:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.898320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.898335:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.907877:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.907976:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.907978:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.907981:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2694b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.907983:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.908010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.908020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.917775:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.917883:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.917884:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.917887:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2694d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.917889:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.917900:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.917909:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.931101:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.931212:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.931214:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.931217:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2694f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.931219:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.931238:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.931248:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.944790:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.944925:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.944927:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.944929:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26951 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.944932:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.944951:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.944960:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.958786:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.958911:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.958913:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.958915:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26953 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.958918:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.958939:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.958953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.972145:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.972279:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.972280:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.972283:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26955 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.972285:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.972310:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.972321:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.982300:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.982661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.982663:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.982666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26957 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.982668:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.982689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.982701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773266.995843:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773266.995943:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773266.995946:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773266.995948:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26959 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773266.995952:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773266.995971:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773266/real 1521773266]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773278 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773266.995981:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.005310:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.005416:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.005417:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.005420:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2695b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.005422:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.005439:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.005448:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.018621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.018880:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.018882:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.018885:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2695d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.018887:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.018907:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.018921:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.028322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.028433:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.028435:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.028437:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2695f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.028440:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.028455:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.028465:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.041592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.041699:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.041701:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.041704:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26961 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.041706:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.041727:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.041736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.052128:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.052294:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.052296:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.052299:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26963 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.052301:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.052323:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.052334:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.062555:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.062746:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.062748:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.062751:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26965 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.062753:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.062787:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.062796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.081830:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.081950:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.081952:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.081955:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26967 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.081957:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.081980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.081995:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.096496:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.096613:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.096615:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.096618:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26969 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.096622:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.096644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.096654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.110833:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.110954:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.110956:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.110959:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2696b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.110961:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.110984:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.111009:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.125081:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.125228:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.125232:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.125238:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2696d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.125243:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.125269:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.125278:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.138770:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.138898:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.138900:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.138902:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2696f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.138905:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.138927:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.138937:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.152371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.152491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.152493:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.152496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26971 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.152498:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.152538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.152554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.162420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.162547:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.162551:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.162557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26973 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.162562:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.162588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.162599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.173321:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.173459:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.173463:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.173469:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26975 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.173474:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.173499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.173531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.187092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.187196:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.187198:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.187200:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26977 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.187203:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.187222:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.187231:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.200868:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.200990:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.200992:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.200994:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26979 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.200998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.201019:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.201028:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.210986:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.211105:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.211107:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.211110:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2697b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.211112:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.211128:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.211143:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.220631:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.220731:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.220732:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.220735:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2697d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.220737:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.220753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.220763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.233780:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.233902:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.233904:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.233907:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2697f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.233909:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.233929:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.233938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.243306:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.243413:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.243415:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.243417:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26981 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.243420:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.243445:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.243455:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.256764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.256879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.256881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.256884:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26983 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.256886:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.256905:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.256914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.266586:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.266691:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.266693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.266696:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26985 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.266698:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.266716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.266731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.276853:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.276972:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.276974:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.276976:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26987 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.276979:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.276994:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.277003:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.287667:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.287785:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.287787:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.287790:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26989 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.287794:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.287810:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.287819:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.301151:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.301275:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.301277:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.301280:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2698b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.301282:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.301301:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.301310:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.310914:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.311035:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.311037:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.311039:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2698d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.311042:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.311057:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.311066:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.320757:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.320869:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.320871:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.320874:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2698f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.320876:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.320891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.320906:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.330744:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.330844:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.330847:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.330849:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26991 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.330852:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.330863:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.330873:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.340536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.340666:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.340670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.340676:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26993 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.340681:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.340705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.340714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.354133:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.354266:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.354268:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.354271:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26995 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.354273:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.354292:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.354301:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.367847:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.367957:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.367959:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.367962:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26997 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.367965:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.367980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.367989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.381186:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.381305:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.381307:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.381310:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26999 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.381314:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.381333:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.381377:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.394621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.394732:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.394734:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.394737:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2699b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.394739:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.394760:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.394785:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.408399:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.408528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.408530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.408533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2699d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.408535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.408562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.408574:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.418593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.418702:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.418704:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.418707:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2699f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.418709:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.418725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.418734:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.432311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.432414:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.432416:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.432419:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269a1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.432421:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.432442:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.432452:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.441421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.441521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.441523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.441525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269a3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.441528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.441539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.441554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.455693:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.455806:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.455808:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.455810:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269a5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.455813:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.455833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.455842:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.465588:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.465695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.465697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.465699:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269a7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.465702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.465718:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.465728:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.472207:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.472794:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.472796:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.472815:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269a9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.472818:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.472874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.472885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.482415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.482519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.482521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.482524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269ab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.482527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.482545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.482555:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.492467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.492604:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.492608:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.492614:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269ad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.492619:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.492646:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.492666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.506556:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.506668:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.506670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.506673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269af sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.506675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.506692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.506702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.520467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.520672:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.520674:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.520677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269b1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.520679:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.520697:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.520707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.535470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.535584:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.535586:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.535589:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269b3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.535592:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.535609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.535619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.549989:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.550101:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.550103:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.550105:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269b5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.550108:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.550124:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.550134:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.564041:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.564161:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.564163:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.564166:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269b7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.564168:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.564185:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.564199:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.578634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.578736:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.578738:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.578741:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269b9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.578745:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.578756:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.578766:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.593065:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.593184:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.593186:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.593188:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269bb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.593190:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.593205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.593214:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.607658:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.607763:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.607765:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.607783:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269bd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.607785:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.607817:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.607827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.621281:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.621408:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.621410:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.621413:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269bf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.621416:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.621427:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.621437:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.635776:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.635895:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.635897:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.635900:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269c1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.635902:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.635918:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.635932:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.650336:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.650450:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.650452:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.650455:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269c3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.650457:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.650472:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.650482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.664085:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.664187:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.664189:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.664191:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269c5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.664194:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.664210:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.664219:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.678115:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.678217:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.678219:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.678222:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269c7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.678225:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.678245:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.678255:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.691498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.691608:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.691610:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.691613:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269c9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.691617:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.691639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.691650:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.706134:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.706248:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.706251:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.706253:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269cb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.706255:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.706270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.706285:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.720379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.720482:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.720484:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.720487:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269cd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.720490:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.720505:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.720525:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.734444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.734542:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.734544:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.734547:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269cf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.734550:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.734568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.734578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.749006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.749108:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.749110:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.749113:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269d1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.749115:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.749130:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.749139:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.762882:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.762995:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.762997:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.762999:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269d3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.763002:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.763017:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.763027:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.776817:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.776934:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.776936:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.776939:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269d5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.776941:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.776957:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.776971:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.791107:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.791214:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.791216:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.791219:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269d7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.791221:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.791238:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.791247:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.805467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.805563:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.805565:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.805568:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269d9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.805572:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.805590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.805600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.819817:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.819922:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.819924:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.819926:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269db sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.819928:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.819945:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.819953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.834062:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.834172:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.834174:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.834177:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269dd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.834179:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.834196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.834205:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.848452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.848557:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.848559:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.848562:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269df sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.848564:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.848582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.848597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.862184:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.862295:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.862297:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.862300:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269e1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.862302:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.862317:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.862326:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.876685:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.876820:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.876822:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.876825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269e3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.876827:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.876843:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.876852:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.890920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.891015:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.891017:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.891020:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269e5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.891022:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.891038:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.891047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.905193:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.905295:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.905297:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.905300:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269e7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.905302:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.905317:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.905326:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.918819:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.918932:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.918949:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.918952:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269e9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.918955:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.918971:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.918985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.933536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.933636:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.933638:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.933641:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269eb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.933643:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.933661:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.933671:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.947430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.947529:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.947531:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.947533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269ed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.947536:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.947553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.947563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.961909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.962010:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.962012:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.962015:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269ef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.962017:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.962032:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.962042:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.975665:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.975763:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.975765:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.975783:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.975785:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.975816:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.975825:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773267.991920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773267.992017:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773267.992019:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773267.992022:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773267.992024:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773267.992041:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773267/real 1521773267]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773279 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773267.992055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.005728:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.005824:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.005826:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.005828:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.005831:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.005841:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.005850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.020214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.020318:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.020320:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.020322:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.020324:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.020339:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.020378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.028423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.028520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.028522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.028525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.028528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.028540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.028550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.041920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.042044:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.042046:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.042049:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.042051:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.042067:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.042076:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.055459:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.055613:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.055618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.055624:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.055629:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.055662:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.055678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.068873:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.068984:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.068987:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.068989:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x269ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.068992:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.069007:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.069015:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.078434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.078523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.078525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.078528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.078530:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.078541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.078551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.088117:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.088212:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.088214:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.088217:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a03 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.088219:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.088245:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.088253:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.097959:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.098062:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.098064:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.098067:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.098069:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.098083:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.098093:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.111423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.111520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.111522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.111532:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.111534:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.111549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.111565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.121450:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.121547:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.121549:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.121552:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.121556:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.121570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.121579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.131426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.131553:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.131555:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.131557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a0b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.131560:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.131581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.131592:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.141029:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.141163:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.141168:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.141173:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.141179:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.141201:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.141210:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.154747:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.154854:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.154856:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.154859:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.154861:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.154882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.154892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.164391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.164489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.164491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.164493:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.164495:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.164517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.164532:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.174117:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.174220:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.174221:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.174224:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.174226:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.174239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.174263:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.187655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.187766:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.187768:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.187771:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a15 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.187773:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.187805:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.187814:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.201659:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.201763:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.201765:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.201784:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.201786:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.201804:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.201813:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.215394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.215535:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.215537:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.215540:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.215544:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.215561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.215571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.225975:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.226066:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.226068:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.226071:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.226073:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.226090:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.226104:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.236227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.236336:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.236338:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.236341:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.236343:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.236371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.236382:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.246022:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.246136:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.246138:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.246140:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.246142:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.246157:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.246167:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.259783:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.259981:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.259983:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.259986:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.259988:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.260004:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.260013:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.269893:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.270003:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.270005:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.270008:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.270010:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.270026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.270035:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.283739:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.283847:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.283849:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.283852:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.283854:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.283871:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.283886:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.298569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.298680:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.298682:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.298684:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.298687:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.298703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.298713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.308605:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.308707:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.308710:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.308713:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.308716:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.308733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.308743:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.322394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.322498:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.322500:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.322503:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.322515:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.322534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.322544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.332910:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.333020:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.333022:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.333025:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.333027:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.333042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.333051:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.343247:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.343343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.343355:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.343358:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.343360:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.343378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.343393:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.356924:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.357040:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.357042:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.357045:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.357047:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.357062:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.357072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.366591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.366689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.366691:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.366693:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.366696:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.366706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.366716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.376628:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.376746:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.376748:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.376751:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.376753:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.376785:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.376794:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.390412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.390519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.390521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.390524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.390526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.390543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.390553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.400214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.400538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.400540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.400542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.400546:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.400567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.400585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.410262:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.410364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.410366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.410368:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.410371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.410385:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.410397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.423644:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.423751:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.423753:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.423756:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.423758:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.423795:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.423804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.433639:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.433739:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.433742:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.433744:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.433747:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.433763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.433773:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.443183:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.443282:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.443284:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.443287:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.443290:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.443324:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.443334:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.456929:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.457035:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.457037:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.457041:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.457043:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.457059:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.457073:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.466742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.466874:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.466876:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.466879:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.466881:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.466892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.466901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.476211:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.476303:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.476305:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.476308:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.476311:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.476322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.476331:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.486262:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.486363:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.486365:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.486367:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.486371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.486391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.486403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.499863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.499961:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.499963:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.499966:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.499968:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.499997:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.500006:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.513190:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.513308:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.513310:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.513312:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.513315:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.513334:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.513364:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.522526:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.522631:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.522633:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.522636:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.522638:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.522650:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.522659:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.535798:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.535903:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.535905:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.535908:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.535910:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.535929:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.535938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.545200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.545313:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.545315:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.545317:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.545320:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.545336:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.545360:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.558590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.558693:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.558695:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.558698:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.558701:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.558722:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.558732:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.572441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.572550:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.572552:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.572555:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.572558:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.572582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.572598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.582212:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.582312:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.582314:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.582317:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.582320:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.582335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.582344:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.592535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.592649:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.592651:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.592654:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.592656:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.592668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.592677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.605761:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.605895:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.605897:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.605899:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.605902:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.605921:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.605931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.615481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.615625:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.615629:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.615635:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.615652:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.615676:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.615686:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.626176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.626297:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.626299:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.626301:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.626303:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.626318:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.626332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.639823:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.639921:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.639923:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.639926:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.639929:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.639948:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.639957:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.653311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.653410:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.653412:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.653415:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.653418:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.653440:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.653450:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.666600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.666710:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.666713:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.666715:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.666718:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.666740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.666749:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.681018:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.681129:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.681131:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.681134:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.681138:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.681157:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.681166:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.689838:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.689943:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.689945:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.689948:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.689950:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.689971:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.689985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.698382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.698495:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.698497:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.698499:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.698502:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.698539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.698549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.707942:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.708073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.708075:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.708078:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.708080:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.708099:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.708108:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.716454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.716566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.716568:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.716571:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.716574:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.716599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.716609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.725131:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.725240:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.725242:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.725245:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.725248:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.725283:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.725308:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.733700:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.733823:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.733825:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.733828:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.733830:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.733854:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.733869:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.742436:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.742554:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.742556:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.742559:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.742562:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.742579:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.742589:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.750892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.751018:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.751020:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.751023:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.751027:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.751046:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.751056:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.760119:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.760220:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.760222:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.760225:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.760227:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.760246:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.760256:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.769089:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.769194:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.769196:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.769198:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.769201:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.769221:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.769230:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.778850:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.778974:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.778976:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.778978:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.778981:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.778996:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.779010:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.787570:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.787681:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.787683:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.787686:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.787689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.787710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.787720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.796452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.796575:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.796577:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.796580:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.796582:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.796599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.796609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.805309:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.805439:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.805441:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.805444:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.805446:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.805469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.805479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.814317:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.814442:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.814444:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.814447:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.814450:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.814468:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.814478:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.823147:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.823250:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.823252:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.823255:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.823258:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.823278:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.823292:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.832406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.832676:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.832679:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.832681:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.832684:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.832706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.832715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.841578:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.841822:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.841824:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.841826:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.841829:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.841847:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.841856:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.850418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.850528:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.850530:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.850533:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.850535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.850565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.850575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.860610:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.860715:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.860717:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.860720:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.860722:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.860739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.860749:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.871302:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.871430:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.871432:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.871435:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.871437:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.871453:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.871468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.880342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.880473:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.880475:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.880478:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.880480:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.880501:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.880533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.889286:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.889409:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.889411:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.889414:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.889416:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.889433:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.889442:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.899342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.899481:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.899483:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.899486:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.899490:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.899531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.899541:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.908381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.908504:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.908510:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.908513:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.908516:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.908549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.908559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.918283:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.918396:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.918397:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.918400:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.918402:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.918419:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.918434:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.927537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.927642:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.927644:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.927647:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26a9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.927649:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.927669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.927678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.936441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.936549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.936551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.936554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aa1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.936556:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.936576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.936586:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.946278:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.946382:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.946384:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.946386:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aa3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.946389:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.946409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.946421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.955215:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.955358:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.955359:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.955362:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aa5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.955364:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.955377:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.955387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.964408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.964519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.964521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.964524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aa7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.964526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.964546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.964561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.973325:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.973462:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.973464:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.973467:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aa9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.973470:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.973487:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.973497:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.982413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.982523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.982525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.982528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.982530:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.982550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.982559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773268.991403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773268.991505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773268.991513:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773268.991516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773268.991518:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773268.991540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773268.991550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.000283:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.000398:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.000400:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.000403:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aaf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.000405:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.000434:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773268/real 1521773268]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773280 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.000444:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.010040:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.010149:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.010151:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.010154:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ab1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.010156:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.010169:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.010201:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.018791:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.018903:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.018905:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.018908:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ab3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.018910:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.018926:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.018935:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.027577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.027683:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.027685:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.027688:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ab5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.027690:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.027709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.027719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.036318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.036432:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.036434:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.036437:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ab7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.036439:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.036474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.036484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.045176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.045283:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.045285:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.045288:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ab9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.045292:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.045343:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.045378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.053958:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.054086:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.054091:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.054096:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26abb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.054102:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.054122:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.054137:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.062794:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.062902:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.062903:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.062906:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26abd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.062912:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.062937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.062946:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.071810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.071932:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.071934:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.071937:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26abf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.071939:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.071974:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.071984:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.080855:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.081010:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.081014:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.081020:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ac1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.081025:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.081055:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.081064:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.089523:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.089640:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.089642:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.089645:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ac3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.089647:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.089670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.089681:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.099566:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.099696:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.099700:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.099706:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ac5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.099711:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.099741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.099756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.108114:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.108226:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.108228:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.108231:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ac7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.108233:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.108247:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.108257:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.117203:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.117328:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.117330:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.117333:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ac9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.117336:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.117366:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.117376:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.126800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.126933:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.126937:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.126943:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26acb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.126948:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.126972:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.126987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.135781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.135925:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.135930:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.135935:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26acd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.135942:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.135967:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.135977:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.144484:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.144617:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.144619:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.144621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26acf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.144623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.144637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.144652:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.153386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.153532:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.153536:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.153542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ad1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.153547:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.153578:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.153589:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.163267:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.163403:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.163408:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.163420:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ad3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.163425:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.163455:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.163465:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.172223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.172369:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.172373:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.172379:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ad5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.172384:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.172412:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.172423:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.182459:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.182599:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.182601:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.182603:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ad7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.182605:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.182631:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.182641:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.192773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.192874:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.192876:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.192878:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ad9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.192882:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.192901:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.192915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.202196:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.202317:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.202319:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.202322:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26adb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.202324:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.202341:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.202365:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.210976:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.211074:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.211076:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.211079:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26add sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.211081:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.211099:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.211109:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.221057:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.221184:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.221186:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.221188:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26adf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.221191:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.221209:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.221218:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.230062:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.230182:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.230184:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.230187:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ae1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.230189:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.230208:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.230217:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.239244:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.239360:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.239362:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.239365:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ae3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.239382:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.239397:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.239411:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.248246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.248382:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.248384:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.248387:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ae5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.248389:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.248409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.248418:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.257040:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.257143:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.257145:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.257147:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ae7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.257149:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.257166:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.257175:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.266182:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.266306:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.266308:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.266311:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ae9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.266314:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.266331:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.266340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.275106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.275224:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.275226:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.275229:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aeb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.275232:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.275249:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.275258:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.284404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.284504:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.284514:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.284532:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.284534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.284555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.284571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.293579:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.293677:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.293679:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.293681:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.293684:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.293705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.293714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.302917:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.303056:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.303058:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.303060:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26af1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.303062:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.303080:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.303089:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.312320:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.312432:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.312434:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.312436:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26af3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.312439:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.312457:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.312466:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.322418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.322537:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.322539:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.322542:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26af5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.322544:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.322565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.322575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.331366:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.331495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.331497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.331500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26af7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.331502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.331543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.331558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.340279:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.340405:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.340407:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.340410:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26af9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.340413:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.340431:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.340443:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.350457:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.350570:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.350573:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.350575:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26afb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.350578:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.350599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.350610:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.359604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.359709:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.359711:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.359714:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26afd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.359717:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.359736:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.359745:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.368424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.368530:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.368532:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.368535:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26aff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.368537:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.368559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.368570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.377432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.377564:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.377569:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.377574:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.377579:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.377601:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.377616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.386300:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.386437:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.386439:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.386442:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b03 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.386445:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.386464:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.386474:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.395314:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.395427:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.395429:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.395432:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.395434:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.395453:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.395462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.405094:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.405224:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.405226:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.405229:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.405231:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.405249:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.405258:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.413943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.414049:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.414051:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.414054:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.414057:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.414074:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.414083:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.422887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.422997:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.422999:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.423002:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b0b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.423004:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.423022:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.423037:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.431680:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.431804:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.431806:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.431809:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.431811:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.431830:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.431839:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.440335:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.440457:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.440459:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.440461:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.440464:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.440481:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.440490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.446374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.447534:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.447536:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.447539:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.447541:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.447563:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.447573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.456394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.456513:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.456515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.456518:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.456520:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.456538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.456548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.465398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.465518:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.465520:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.465523:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b15 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.465526:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.465546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.465561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.475335:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.475474:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.475476:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.475479:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.475481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.475499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.475532:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.484446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.484559:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.484561:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.484564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.484567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.484590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.484600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.493337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.493531:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.493533:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.493536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.493539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.493559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.493569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.502497:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.502615:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.502617:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.502620:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.502623:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.502635:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.502644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.512564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.512674:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.512676:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.512679:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.512681:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.512700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.512716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.521452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.521573:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.521575:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.521578:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.521580:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.521600:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.521610:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.530376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.530474:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.530476:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.530479:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.530481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.530499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.530516:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.539063:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.539165:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.539167:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.539184:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.539187:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.539205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.539214:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.549053:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.549156:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.549158:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.549160:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.549163:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.549180:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.549189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.557879:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.557990:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.557992:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.557994:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.557998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.558015:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.558030:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.568195:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.568322:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.568324:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.568326:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.568329:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.568361:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.568371:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.578051:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.578170:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.578172:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.578175:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.578177:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.578195:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.578205:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.587031:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.587137:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.587139:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.587157:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.587160:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.587179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.587188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.596404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.596517:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.596519:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.596522:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.596524:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.596543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.596553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.605197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.605317:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.605319:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.605322:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.605324:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.605341:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.605364:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.614760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.614887:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.614889:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.614891:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.614893:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.614911:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.614921:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.623776:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.623880:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.623882:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.623885:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.623887:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.623906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.623915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.632911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.633014:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.633017:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.633019:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.633023:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.633040:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.633050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.642971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.643078:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.643080:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.643083:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.643086:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.643102:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.643112:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.653540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.653647:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.653649:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.653652:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.653655:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.653672:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.653687:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.663742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.663860:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.663862:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.663864:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.663867:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.663878:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.663887:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.674178:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.674287:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.674289:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.674292:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.674295:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.674310:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.674319:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.685618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.685731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.685733:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.685736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.685738:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.685755:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.685765:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.695954:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.696068:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.696070:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.696073:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.696075:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.696091:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.696100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.706415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.706529:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.706531:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.706534:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.706536:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.706557:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.706573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.717495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.717617:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.717619:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.717622:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.717626:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.717643:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.717654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.727785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.727901:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.727903:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.727906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.727908:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.727924:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.727933:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.738133:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.738241:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.738243:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.738245:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.738248:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.738262:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.738271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.748589:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.748694:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.748696:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.748699:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.748702:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.748721:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.748731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.758882:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.758992:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.758994:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.758997:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.758999:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.759017:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.759032:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.769106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.769208:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.769210:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.769212:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.769215:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.769248:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.769273:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.779563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.779679:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.779681:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.779684:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.779686:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.779706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.779716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.789936:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.790047:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.790049:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.790052:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.790054:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.790073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.790082:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.800285:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.800393:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.800395:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.800398:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.800401:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.800418:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.800430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.810456:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.810568:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.810570:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.810573:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.810576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.810596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.810611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.821880:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.821984:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.821986:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.821989:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.821991:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.822008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.822018:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.832596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.832707:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.832710:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.832712:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.832715:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.832734:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.832744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.843367:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.843469:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.843471:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.843474:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.843476:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.843494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.843503:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.853625:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.853739:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.853741:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.853744:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.853746:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.853765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.853790:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.864110:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.864220:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.864222:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.864225:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.864227:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.864244:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.864258:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.874528:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.874635:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.874637:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.874640:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.874643:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.874663:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.874674:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.884691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.884806:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.884808:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.884811:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.884814:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.884833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.884842:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.918164:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.918290:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.918292:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.918294:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.918297:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.918316:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.918325:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.928705:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.928811:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.928813:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.928816:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.928834:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.928867:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.928876:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.939429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.939565:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.939567:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.939570:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.939572:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.939596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.939613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.950671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.950790:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.950792:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.950794:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.950797:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.950816:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.950826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.961065:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.961189:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.961191:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.961194:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.961196:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.961213:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.961222:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.971407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.971516:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.971518:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.971521:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.971523:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.971545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.971555:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.982762:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.982887:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.982889:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.982892:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.982894:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.982911:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.982920:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773269.993461:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773269.993571:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773269.993573:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773269.993576:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773269.993579:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773269.993597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773269/real 1521773269]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773281 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773269.993612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.003704:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.003809:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.003811:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.003830:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.003832:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.003851:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.003860:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.014020:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.014160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.014164:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.014183:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.014185:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.014209:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.014219:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.024448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.024576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.024578:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.024581:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.024583:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.024608:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.024619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.034980:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.035128:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.035132:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.035138:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.035143:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.035173:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.035182:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.045325:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.045477:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.045479:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.045482:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.045484:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.045531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.045547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.055530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.055634:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.055636:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.055639:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.055641:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.055661:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.055671:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.065446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.065559:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.065562:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.065564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.065567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.065587:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.065597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.075399:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.075519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.075522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.075524:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.075528:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.075550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.075560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.085615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.085725:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.085728:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.085730:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.085733:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.085745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.085754:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.095591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.095703:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.095705:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.095708:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.095711:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.095730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.095745:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.105821:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.105930:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.105932:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.105935:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.105937:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.105971:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.105981:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.116942:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.117037:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.117039:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.117042:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.117044:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.117063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.117072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.132541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.132653:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.132655:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.132658:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.132660:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.132684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.132695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.142604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.142714:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.142717:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.142719:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.142722:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.142737:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.142746:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.153184:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.153307:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.153309:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.153311:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.153314:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.153331:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.153361:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.163489:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.163647:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.163649:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.163652:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.163656:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.163675:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.163685:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.173912:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.174029:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.174031:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.174033:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.174036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.174052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.174061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.184071:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.184175:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.184177:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.184180:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.184182:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.184215:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.184240:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.194418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.194542:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.194544:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.194547:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26b9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.194549:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.194569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.194579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.204485:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.204601:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.204603:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.204606:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ba1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.204608:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.204621:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.204636:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.214875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.214984:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.214986:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.214988:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ba3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.214990:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.215002:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.215011:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.226403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.226530:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.226532:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.226535:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ba5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.226537:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.226555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.226565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.237774:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.237883:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.237885:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.237888:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ba7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.237890:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.237909:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.237918:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.248155:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.248268:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.248270:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.248273:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ba9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.248276:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.248294:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.248303:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.259522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.259638:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.259640:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.259643:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.259645:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.259658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.259672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.269610:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.269716:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.269718:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.269721:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.269723:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.269742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.269751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.279701:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.279800:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.279802:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.279805:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26baf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.279807:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.279819:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.279844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.289644:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.289750:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.289752:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.289755:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bb1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.289757:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.289791:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.289800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.299635:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.299743:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.299745:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.299747:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bb3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.299750:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.299784:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.299793:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.310731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.310845:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.310847:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.310850:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bb5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.310852:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.310871:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.310885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.320674:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.320794:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.320796:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.320799:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bb7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.320801:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.320819:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.320829:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.331387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.331495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.331497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.331500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bb9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.331504:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.331542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.331552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.342074:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.342189:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.342191:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.342193:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bbb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.342195:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.342213:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.342222:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.353039:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.353149:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.353151:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.353154:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bbd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.353156:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.353190:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.353200:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.363450:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.363552:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.363554:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.363556:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bbf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.363559:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.363583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.363599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.373585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.373691:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.373693:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.373695:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bc1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.373698:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.373716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.373726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.383994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.384097:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.384099:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.384117:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bc3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.384119:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.384137:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.384146:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.391086:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.391231:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.391233:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.391235:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bc5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.391238:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.391254:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.391264:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.401432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.401544:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.401546:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.401549:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bc7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.401551:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.401573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.401583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.411655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.411755:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.411758:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.411760:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bc9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.411764:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.411776:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.411791:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.422944:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.423053:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.423055:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.423057:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bcb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.423059:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.423078:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.423087:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.433330:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.433471:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.433473:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.433475:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bcd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.433478:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.433495:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.433504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.446537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.446644:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.446647:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.446649:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bcf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.446652:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.446664:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.446674:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.457142:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.457264:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.457266:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.457268:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bd1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.457271:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.457287:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.457296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.467585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.467693:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.467695:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.467698:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bd3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.467700:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.467735:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.467749:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.478134:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.478586:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.478588:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.478591:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bd5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.478593:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.478615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.478625:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.488931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.489034:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.489036:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.489039:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bd7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.489041:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.489052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.489061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.499117:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.499239:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.499241:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.499244:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bd9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.499247:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.499267:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.499276:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.510217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.510322:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.510324:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.510327:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bdb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.510329:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.510374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.510384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.521542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.521653:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.521655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.521657:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bdd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.521660:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.521679:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.521694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.532498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.532613:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.532615:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.532617:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bdf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.532620:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.532651:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.532662:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.543188:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.543302:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.543304:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.543307:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26be1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.543309:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.543326:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.543335:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.554472:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.561733:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.561735:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.561738:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26be3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.561740:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.561785:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.561796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.572504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.572627:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.572630:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.572632:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26be5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.572634:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.572661:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.572672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.582700:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.582811:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.582814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.582816:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26be7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.582819:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.582853:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.582867:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.593123:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.593252:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.593254:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.593257:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26be9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.593260:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.593278:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.593287:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.601375:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.601485:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.601487:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.601490:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26beb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.601492:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.601532:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.601542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.611740:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.611840:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.611842:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.611845:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.611847:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.611866:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.611876:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.623158:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.623265:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.623267:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.623270:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.623272:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.623290:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.623298:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.633768:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.633917:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.633919:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.633922:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bf1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.633924:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.633943:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.633958:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.644505:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.644629:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.644631:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.644634:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bf3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.644636:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.644654:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.644664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.655215:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.655323:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.655325:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.655327:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bf5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.655329:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.655376:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.655386:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.665468:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.665563:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.665565:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.665568:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bf7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.665570:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.665588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.665597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.675462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.675567:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.675569:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.675572:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bf9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.675576:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.675593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.675603:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.686337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.686455:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.686457:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.686460:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bfb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.686462:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.686477:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.686492:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.697764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.697894:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.697896:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.697899:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bfd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.697901:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.697915:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.697924:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.707773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.707890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.707892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.707894:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26bff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.707896:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.707912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.707921:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.725152:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.725279:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.725281:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.725284:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.725286:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.725302:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.725311:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.735587:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.735699:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.735701:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.735704:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c03 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.735706:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.735723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.735733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.745627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.745724:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.745726:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.745729:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.745731:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.745747:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.745762:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.755490:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.755602:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.755604:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.755607:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.755609:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.755621:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.755631:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.765609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.765718:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.765720:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.765723:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.765726:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.765746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.765756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.776954:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.777074:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.777076:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.777079:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c0b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.777081:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.777098:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.777107:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.788029:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.788137:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.788140:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.788142:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.788145:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.788179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.788188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.798454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.798552:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.798555:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.798557:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.798560:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.798580:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.798595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.809262:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.809372:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.809374:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.809377:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.809379:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.809398:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.809408:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.819614:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.819725:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.819727:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.819730:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.819732:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.819751:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.819761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.829907:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.830010:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.830012:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.830014:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c15 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.830017:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.830034:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.830044:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.840649:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.840753:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.840756:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.840758:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.840761:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.840795:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.840805:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.850722:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.850848:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.850850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.850853:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.850856:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.850875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.850891:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.861557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.861683:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.861685:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.861688:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.861690:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.861711:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.861721:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.871677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.871790:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.871808:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.871811:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.871813:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.871831:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.871841:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.883066:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.883194:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.883196:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.883198:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.883201:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.883218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.883227:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.893650:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.893760:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.893762:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.893765:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.893782:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.893801:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.893811:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.903980:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.904095:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.904097:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.904100:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.904102:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.904121:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.904136:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.914318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.914442:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.914444:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.914447:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.914449:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.914469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.914479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.925187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.925296:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.925298:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.925316:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.925318:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.925335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.925344:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.936501:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.936641:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.936643:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.936646:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.936649:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.936669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.936679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.947810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.947911:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.947913:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.947916:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.947918:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.947931:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.947940:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.957975:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.958093:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.958095:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.958098:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.958100:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.958118:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.958133:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.968491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.968604:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.968606:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.968609:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.968611:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.968631:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.968641:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.979764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.979881:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.979883:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.979885:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.979888:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.979904:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.979914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773270.990022:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773270.990137:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773270.990139:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773270.990141:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773270.990144:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773270.990162:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773270.990172:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.000606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.000706:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.000708:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.000711:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.000713:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.000733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773270/real 1521773270]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773282 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.000742:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.011772:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.011881:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.011883:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.011885:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.011888:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.011906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.011921:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.022478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.022615:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.022620:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.022625:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.022633:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.022672:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.022683:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.033340:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.033474:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.033476:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.033479:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.033481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.033503:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.033537:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.043632:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.043757:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.043759:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.043762:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.043765:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.043870:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.043881:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.054617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.054725:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.054727:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.054730:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.054733:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.054752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.054762:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.064913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.065042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.065044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.065047:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.065049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.065067:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.065082:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.074976:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.075077:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.075079:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.075082:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.075084:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.075102:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.075111:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.084901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.085040:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.085042:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.085044:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.085046:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.085064:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.085073:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.095564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.095692:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.095694:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.095697:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.095699:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.095713:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.095723:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.105524:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.105641:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.105643:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.105646:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.105650:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.105663:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.105672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.116540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.116646:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.116648:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.116651:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.116654:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.116673:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.116688:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.126407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.126534:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.126536:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.126538:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.126541:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.126558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.126568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.136407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.136543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.136545:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.136548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.136550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.136566:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.136575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.147872:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.147988:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.147990:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.147992:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.147995:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.148007:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.148016:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.158438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.158541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.158543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.158545:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.158548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.158571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.158582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.169461:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.169580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.169582:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.169585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.169587:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.169610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.169626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.179531:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.179660:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.179662:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.179665:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.179667:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.179680:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.179689:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.190388:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.190505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.190514:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.190532:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.190536:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.190555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.190565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.201931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.202080:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.202082:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.202085:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.202087:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.202105:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.202114:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.213521:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.213627:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.213629:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.213632:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.213634:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.213647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.213656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.223602:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.223710:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.223713:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.223715:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.223718:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.223737:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.223752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.233943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.234054:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.234056:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.234059:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.234061:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.234078:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.234088:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.243945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.244061:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.244063:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.244066:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.244068:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.244086:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.244095:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.254870:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.254988:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.254990:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.254993:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.254995:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.255012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.255021:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.265909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.266021:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.266023:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.266025:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.266028:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.266046:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.266055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.276420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.276543:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.276545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.276547:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.276551:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.276575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.276593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.286870:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.286991:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.286993:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.286995:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.286998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.287015:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.287025:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.296562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.296683:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.296685:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.296687:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.296690:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.296708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.296717:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.307366:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.307513:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.307515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.307534:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.307536:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.307556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.307566:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.317328:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.317463:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.317465:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.317468:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.317470:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.317488:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.317497:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.327373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.327475:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.327477:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.327480:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.327482:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.327500:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.327540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.337916:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.338016:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.338018:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.338021:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.338023:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.338062:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.338071:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.347868:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.347992:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.347994:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.347997:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.347999:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.348011:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.348020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.355115:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.355248:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.355250:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.355253:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.355256:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.355274:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.355284:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.365183:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.365280:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.365282:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.365285:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.365287:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.365323:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.365378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.375568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.375665:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.375667:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.375670:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.375672:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.375693:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.375707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.386087:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.386201:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.386203:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.386206:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.386208:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.386236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.386245:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.396773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.396891:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.396893:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.396896:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.396898:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.396910:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.396919:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.406892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.407010:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.407012:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.407015:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.407017:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.407029:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.407038:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.417566:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.417673:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.417675:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.417677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.417680:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.417698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.417707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.427564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.427669:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.427672:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.427674:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.427677:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.427695:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.427711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.438379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.438490:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.438492:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.438494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.438498:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.438519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.438528:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.448966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.449101:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.449103:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.449105:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.449123:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.449140:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.449149:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.459303:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.459423:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.459425:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.459428:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.459431:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.459452:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.459463:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.469281:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.469402:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.469404:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.469406:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.469409:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.469429:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.469440:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.480406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.480519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.480521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.480524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.480526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.480547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.480562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.491444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.491553:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.491555:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.491557:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.491560:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.491582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.491592:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.502941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.503058:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.503060:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.503063:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.503065:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.503084:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.503093:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.513437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.513546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.513548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.513551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.513553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.513573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.513583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.524423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.524530:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.524532:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.524535:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.524539:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.524562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.524573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.534385:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.534494:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.534496:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.534498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.534501:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.534547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.534563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.545215:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.545321:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.545323:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.545326:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.545328:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.545359:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.545385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.555804:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.555915:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.555917:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.555920:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26c9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.555922:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.555960:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.555985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.566575:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.566675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.566677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.566680:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ca1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.566682:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.566694:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.566704:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.577403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.577520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.577522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.577525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ca3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.577527:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.577550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.577560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.587973:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.588086:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.588088:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.588091:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ca5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.588093:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.588111:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.588125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.598024:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.598134:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.598136:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.598139:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ca7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.598141:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.598159:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.598169:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.608911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.609023:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.609025:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.609028:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ca9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.609031:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.609047:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.609057:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.619809:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.619905:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.619907:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.619909:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.619911:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.619930:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.619939:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.630398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.630499:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.630501:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.630504:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.630515:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.630528:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.630538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.641541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.641646:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.641648:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.641651:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26caf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.641653:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.641672:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.641687:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.652343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.652469:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.652471:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.652474:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cb1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.652476:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.652492:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.652502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.662212:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.662323:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.662325:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.662328:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cb3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.662330:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.662344:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.662385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.672430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.672546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.672548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.672551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cb5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.672553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.672575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.672585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.681896:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.682017:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.682019:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.682022:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cb7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.682024:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.682039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.682049:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.691636:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.691745:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.691747:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.691750:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cb9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.691754:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.691765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.691795:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.701623:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.701731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.701733:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.701736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cbb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.701738:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.701754:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.701763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.711839:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.711951:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.711953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.711971:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cbd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.711973:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.711983:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.711992:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.722318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.722421:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.722423:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.722426:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cbf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.722428:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.722458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.722468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.732454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.732556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.732558:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.732561:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cc1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.732563:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.732575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.732585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.742435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.742547:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.742549:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.742552:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cc3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.742554:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.742576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.742593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.752278:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.752394:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.752396:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.752399:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cc5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.752401:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.752424:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.752435:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.762107:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.762210:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.762212:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.762214:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cc7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.762217:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.762232:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.762241:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.771618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.771724:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.771726:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.771729:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cc9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.771733:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.771743:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.771752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.781555:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.781659:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.781661:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.781664:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ccb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.781666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.781683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.781692:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.792148:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.792251:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.792253:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.792256:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ccd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.792258:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.792273:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.792288:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.802327:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.802458:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.802460:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.802462:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ccf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.802465:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.802480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.802489:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.812492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.812602:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.812604:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.812607:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cd1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.812609:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.812627:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.812637:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.821948:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.822062:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.822064:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.822066:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cd3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.822068:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.822084:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.822093:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.832163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.832268:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.832270:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.832273:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cd5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.832275:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.832291:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.832300:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.842689:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.842799:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.842801:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.842804:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cd7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.842806:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.842821:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.842836:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.853060:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.853175:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.853177:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.853180:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cd9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.853183:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.853198:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.853207:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.863232:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.863340:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.863342:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.863355:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cdb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.863358:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.863377:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.863388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.873129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.873227:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.873230:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.873232:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cdd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.873235:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.873270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.873279:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.883414:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.883525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.883527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.883530:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cdf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.883532:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.883554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.883564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.893282:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.893409:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.893411:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.893414:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ce1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.893416:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.893433:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.893450:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.903716:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.903820:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.903822:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.903825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ce3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.903827:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.903843:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.903852:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.913749:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.913869:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.913871:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.913874:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ce5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.913876:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.913891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.913900:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.923527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.923632:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.923634:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.923637:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ce7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.923639:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.923655:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.923664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.933186:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.933343:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.933373:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.933376:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ce9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.933398:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.933418:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.933429:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.943231:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.943366:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.943368:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.943370:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ceb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.943373:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.943386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.943404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.953684:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.953790:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.953792:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.953795:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ced sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.953797:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.953814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.953824:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.963762:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.963883:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.963885:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.963888:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.963890:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.963905:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.963915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.974342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.974483:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.974485:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.974488:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cf1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.974491:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.974516:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.974542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.983622:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.983727:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.983729:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.983732:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cf3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.983734:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.983745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.983754:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773271.993448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773271.993555:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773271.993557:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773271.993560:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cf5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773271.993562:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773271.993581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773271/real 1521773271]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773283 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773271.993597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.004025:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.004154:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.004156:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.004159:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cf7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.004161:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.004178:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.004204:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.014212:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.014332:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.014334:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.014337:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cf9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.014340:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.014386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.014396:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.024362:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.024478:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.024480:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.024482:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cfb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.024485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.024500:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.024534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.034438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.034539:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.034541:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.034544:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cfd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.034546:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.034558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.034568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.044309:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.044411:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.044413:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.044416:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26cff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.044418:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.044430:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.044460:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.054575:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.054682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.054684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.054687:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.054689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.054706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.054716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.064578:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.064658:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.064662:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d03 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773272.064664:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.064682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.064691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.074223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.074340:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.074342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.074344:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.074356:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.074418:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.074428:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.084221:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.084359:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.084361:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.084364:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.084366:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.084383:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.084392:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.094423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.094556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.094560:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.094566:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.094571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.094597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.094613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.104384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.104477:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.104481:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d0b sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773272.104483:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.104504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.104540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.114275:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.114418:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.114422:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.114428:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.114433:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.114459:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.114470:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.124491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.124610:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.124612:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.124614:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.124617:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.124641:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.124651:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.133911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.134047:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.134051:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.134056:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.134061:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.134086:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.134096:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.143853:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.143971:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.143973:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.143975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.143977:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.143992:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.144007:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.154864:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.154947:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.154950:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d15 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773272.154953:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.154981:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.154991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.165763:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.165878:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.165880:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.165883:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.165885:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.165900:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.165910:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.176453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.176615:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.176620:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.176626:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.176631:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.176664:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.176675:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.187157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.187273:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.187275:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.187279:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.187281:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.187304:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.187314:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.197911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.198029:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.198031:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.198034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.198036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.198053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.198067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.209395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.209502:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.209504:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.209513:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.209531:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.209550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.209560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.220059:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.220172:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.220174:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.220176:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.220179:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.220196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.220205:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.230841:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.230941:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.230943:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.230946:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.230948:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.230966:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.230975:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.241757:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.241884:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.241885:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.241888:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.241890:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.241908:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.241917:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.252532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.252651:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.252653:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.252655:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.252658:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.252671:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.252685:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.263336:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.263485:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.263487:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.263490:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.263492:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.263533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.263543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.274734:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.274867:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.274869:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.274873:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.274875:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.274892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.274901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.285457:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.285577:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.285579:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.285582:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.285584:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.285606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.285616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.295956:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.296077:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.296079:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.296082:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.296084:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.296102:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.296111:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.307525:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.307638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.307640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.307643:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.307645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.307658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.307673:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.315150:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.315257:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.315259:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.315261:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.315264:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.315281:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.315291:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.325214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.325332:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.325334:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.325336:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.325339:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.325365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.325375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.336309:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.336418:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.336420:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.336423:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.336425:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.336445:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.336455:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.346402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.346557:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.346559:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.346562:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.346564:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.346590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.346602:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.358034:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.358159:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.358161:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.358165:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.358167:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.358186:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.358200:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.368504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.368616:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.368618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.368621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.368623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.368636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.368645:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.379593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.379695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.379697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.379700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.379702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.379721:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.379731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.390382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.390492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.390494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.390497:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.390499:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.390518:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.390544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.406126:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.406248:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.406250:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.406252:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.406255:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.406272:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.406281:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.416592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.416702:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.416704:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.416707:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.416710:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.416730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.416745:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.429414:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.429546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.429548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.429551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.429553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.429573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.429583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.439590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.439695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.439697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.439700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.439703:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.439722:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.439732:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.449627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.449753:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.449755:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.449759:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.449761:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.449796:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.449804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.460006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.460134:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.460136:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.460139:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.460141:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.460159:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.460169:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.470612:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.470716:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.470718:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.470720:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.470723:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.470742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.470757:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.482107:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.482218:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.482220:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.482223:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.482225:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.482242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.482252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.492102:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.492212:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.492214:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.492217:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.492219:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.492236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.492245:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.502428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.502580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.502582:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.502585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.502588:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.502614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.502626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.512584:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.512682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.512684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.512686:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.512689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.512709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.512719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.524918:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.525021:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.525023:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.525026:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.525028:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.525046:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.525062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.536985:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.537095:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.537097:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.537100:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.537103:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.537121:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.537130:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.547632:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.547733:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.547735:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.547738:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.547740:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.547752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.547762:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.557796:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.558078:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.558080:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.558083:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.558085:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.558101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.558110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.568202:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.568310:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.568312:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.568315:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.568317:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.568335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.568370:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.578593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.578832:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.578834:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.578837:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.578839:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.578857:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.578873:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.589859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.589956:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.589958:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.589961:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.589963:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.589980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.589990:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.599569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.599663:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.599665:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.599668:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.599670:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.599682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.599691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.609630:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.609724:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.609726:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.609728:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.609731:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.609749:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.609759:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.620379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.620502:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.620504:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.620516:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.620534:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.620554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.620564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.630486:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.630598:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.630601:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.630603:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.630606:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.630618:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.630633:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.641789:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.641917:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.641919:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.641922:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.641924:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.641941:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.641950:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.652560:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.652668:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.652670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.652673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.652675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.652692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.652702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.662402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.662525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.662527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.662530:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.662532:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.662556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.662566:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.672585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.672683:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.672686:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.672688:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.672691:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.672701:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.672710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.682464:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.682578:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.682580:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.682582:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.682585:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.682604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.682620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.692552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.692657:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.692659:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.692661:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.692664:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.692681:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.692690:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.702930:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.703040:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.703042:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.703045:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.703048:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.703063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.703072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.712929:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.713041:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.713043:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.713046:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.713048:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.713065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.713074:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.722517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.722620:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.722622:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.722625:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.722628:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.722639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.722648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.732444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.732556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.732558:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.732561:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.732564:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.732583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.732598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.742618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.742732:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.742734:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.742736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.742739:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.742756:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.742766:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.752556:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.752665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.752667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.752670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.752673:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.752684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.752693:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.762468:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.762580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.762582:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.762585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.762588:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.762600:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.762610:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.772559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.772669:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.772671:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.772674:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.772676:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.772692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.772702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.782598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.782733:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.782737:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.782745:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.782750:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.782793:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.782808:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.792782:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.792908:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.792918:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.792920:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.792923:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.792938:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.792947:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.803141:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.803245:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.803247:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.803249:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.803252:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.803267:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.803276:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.812643:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.812752:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.812754:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.812757:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.812759:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.812790:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.812800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.823009:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.823117:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.823119:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.823122:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.823124:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.823139:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.823148:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.832595:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.832699:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.832701:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.832704:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.832706:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.832716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.832731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.842582:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.842686:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.842688:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.842691:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.842693:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.842710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.842720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.852735:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.852835:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.852837:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.852840:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.852843:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.852874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.852883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.863068:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.863188:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.863190:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.863194:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.863197:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.863211:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.863221:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.873376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.873479:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.873481:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.873484:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.873486:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.873504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.873523:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.883166:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.883259:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.883261:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.883264:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26d9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.883266:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.883281:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.883295:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.893467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.893580:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.893582:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.893584:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26da1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.893587:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.893599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.893609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.903810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.903937:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.903939:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.903941:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26da3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.903944:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.903959:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.903968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.913954:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.914058:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.914060:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.914062:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26da5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.914065:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.914081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.914090:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.923695:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.923807:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.923809:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.923812:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26da7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.923814:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.923831:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.923841:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.934456:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.934596:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.934598:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.934601:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26da9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.934604:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.934621:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.934636:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.944430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.944532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.944534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.944538:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.944541:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.944558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.944568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.954493:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.954617:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.954619:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.954622:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.954624:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.954642:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.954652:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.964280:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.964413:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.964415:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.964418:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26daf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.964420:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.964438:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.964450:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.974608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.974721:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.974723:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.974726:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26db1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.974728:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.974746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.974756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.984482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.984600:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.984602:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.984605:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26db3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.984607:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.984618:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.984633:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773272.993645:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773272.993746:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773272.993748:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773272.993751:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26db5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773272.993754:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773272.993765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773272/real 1521773272]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773284 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773272.993775:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.003767:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.003877:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.003879:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.003881:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26db7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.003883:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.003899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.003908:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.013524:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.013638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.013640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.013642:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26db9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.013645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.013658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.013668:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.023082:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.023188:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.023190:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.023193:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dbb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.023196:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.023211:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.023220:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.032968:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.033083:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.033085:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.033087:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dbd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.033090:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.033100:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.033114:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.043311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.043417:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.043419:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.043422:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dbf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.043424:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.043444:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.043455:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.053454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.053561:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.053563:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.053566:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dc1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.053568:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.053592:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.053603:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.063590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.063692:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.063694:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.063697:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dc3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.063699:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.063716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.063726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.073681:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.073787:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.073789:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.073792:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dc5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.073794:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.073826:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.073835:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.084177:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.084281:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.084283:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.084285:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dc7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.084288:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.084303:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.084318:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.094545:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.094657:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.094659:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.094662:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dc9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.094664:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.094681:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.094691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.104385:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.104496:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.104498:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.104502:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dcb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.104505:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.104535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.104545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.110909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.111005:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.111007:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.111010:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dcd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.111012:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.111024:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.111033:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.122568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.122667:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.122669:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.122672:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dcf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.122674:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.122686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.122695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.132147:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.132248:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.132250:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.132253:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dd1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.132255:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.132265:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.132296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.142420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.142571:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.142573:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.142575:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dd3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.142578:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.142602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.142613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.152393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.152519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.152521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.152524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dd5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.152526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.152541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.152551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.161817:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.161933:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.161935:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.161938:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dd7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.161940:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.161967:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.161976:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.171569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.171690:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.171692:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.171694:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dd9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.171696:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.171719:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.171729:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.181394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.181525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.181527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.181531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ddb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.181534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.181563:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.181579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.191608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.191745:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.191749:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.191755:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ddd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.191760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.191798:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.191808:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.201892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.202015:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.202017:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.202019:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ddf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.202022:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.202042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.202052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.211886:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.211998:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.212000:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.212002:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26de1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.212005:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.212018:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.212027:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.221990:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.222104:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.222106:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.222109:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26de3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.222111:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.222131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.222140:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.233197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.233318:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.233320:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.233323:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26de5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.233325:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.233340:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.233365:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.240127:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.240992:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.240994:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.240997:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26de7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.240999:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.241057:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.241068:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.250964:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.251089:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.251091:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.251093:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26de9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.251096:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.251116:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.251125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.261431:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.261550:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.261552:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.261556:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26deb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.261559:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.261582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.261592:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.280165:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.280286:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.280288:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.280291:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ded sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.280293:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.280308:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.280318:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.290268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.290414:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.290416:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.290418:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26def sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.290421:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.290432:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.290447:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.300462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.300606:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.300608:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.300611:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26df1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.300613:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.300641:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.300653:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.310536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.310645:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.310647:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.310650:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26df3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.310652:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.310668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.310677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.321920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.322028:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.322030:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.322032:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26df5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.322035:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.322050:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.322060:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.332148:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.332264:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.332266:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.332268:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26df7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.332271:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.332290:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.332300:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.342090:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.342198:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.342200:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.342203:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26df9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.342205:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.342220:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.342234:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.352702:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.352814:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.352816:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.352820:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dfb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.352823:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.352844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.352853:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.362376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.362490:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.362492:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.362495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dfd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.362497:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.362538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.362548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.372306:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.372413:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.372430:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.372433:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26dff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.372435:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.372452:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.372462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.382282:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.382420:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.382424:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.382430:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.382439:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.382457:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.382477:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.392401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.392538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.392542:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.392548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e03 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.392553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.392588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.392604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.402748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.402859:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.402861:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.402864:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.402866:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.402888:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.402897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.412857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.412980:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.412982:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.412985:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.412987:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.413008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.413017:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.423540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.423638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.423640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.423643:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.423645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.423663:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.423673:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.433488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.433611:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.433613:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.433617:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e0b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.433619:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.433636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.433646:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.443367:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.443478:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.443480:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.443483:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.443485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.443500:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.443539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.453411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.453522:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.453524:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.453527:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.453529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.453555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.453565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.463232:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.463356:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.463375:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.463377:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.463380:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.463395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.463405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.471323:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.471441:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.471444:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.471446:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.471449:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.471458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.471468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.481690:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.481805:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.481807:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.481810:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e15 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.481812:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.481845:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.481854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.491899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.492012:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.492014:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.492017:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.492019:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.492034:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.492049:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.507559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.507666:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.507668:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.507670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.507673:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.507691:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.507700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.517759:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.517862:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.517864:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.517868:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.517870:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.517889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.517898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.527961:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.528058:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.528060:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.528063:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.528081:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.528099:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.528124:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.539663:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.539819:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.539821:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.539824:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.539842:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.539859:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.539868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.548608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.548724:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.548726:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.548729:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.548732:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.548751:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.548773:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.558903:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.559007:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.559009:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.559012:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.559014:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.559033:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.559043:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.569942:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.570051:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.570053:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.570055:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.570058:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.570075:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.570100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.582596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.582722:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.582724:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.582727:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.582729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.582741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.582750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.600237:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.600364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.600366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.600368:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.600371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.600392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.600403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.611593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.611697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.611699:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.611702:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.611705:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.611725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.611740:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.622840:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.622945:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.622947:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.622949:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.622951:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.622970:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.622979:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.633409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.633547:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.633549:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.633552:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.633555:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.633575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.633584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.651267:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.651389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.651391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.651393:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.651396:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.651416:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.651428:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.662931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.663058:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.663059:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.663062:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.663064:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.663079:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.663088:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.673980:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.674102:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.674104:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.674106:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.674109:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.674125:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.674140:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.684250:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.684380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.684382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.684385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.684387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.684404:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.684414:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.700887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.701007:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.701009:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.701011:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.701014:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.701029:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.701038:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.711424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.711541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.711543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.711547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.711550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.711570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.711580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.722322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.722472:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.722474:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.722477:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.722480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.722494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.722504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.733430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.733546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.733548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.733551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.733553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.733571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.733587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.745975:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.746096:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.746098:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.746101:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.746103:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.746119:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.746128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.757246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.757380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.757382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.757385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.757387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.757405:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.757415:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.768246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.768382:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.768384:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.768387:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.768389:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.768406:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.768416:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.778527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.778643:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.778645:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.778648:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.778651:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.778669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.778679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.793956:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.794072:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.794074:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.794077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.794079:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.794096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.794110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.805044:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.805150:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.805152:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.805155:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.805158:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.805174:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.805183:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.815236:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.815380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.815382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.815385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.815387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.815404:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.815414:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.832575:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.832705:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.832707:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.832710:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.832712:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.832728:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.832737:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.843948:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.844059:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.844060:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.844063:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.844065:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.844082:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.844091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.854440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.854560:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.854562:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.854564:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.854567:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.854591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.854607:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.871934:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.872046:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.872048:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.872051:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.872053:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.872069:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.872078:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.882973:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.883293:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.883295:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.883297:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.883299:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.883341:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.883385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.894590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.894713:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.894715:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.894718:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.894720:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.894738:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.894748:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.904837:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.904951:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.904953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.904957:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.904959:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.904976:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.904985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.915927:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.916042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.916044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.916047:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.916049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.916065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.916080:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.926295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.926420:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.926421:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.926424:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.926426:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.926444:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.926456:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.937297:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.937404:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.937406:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.937409:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.937411:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.937432:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.937444:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.947581:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.947692:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.947694:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.947697:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.947699:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.947709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.947719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.958565:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.958664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.958666:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.958669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.958671:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.958683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.958692:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.969470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.969580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.969583:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.969585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.969588:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.969606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.969621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.979421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.979525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.979527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.979529:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.979535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.979561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.979571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773273.989417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773273.989539:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773273.989541:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773273.989545:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773273.989547:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773273.989572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773273.989582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.000451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.000574:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.000576:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.000579:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.000581:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.000599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773273/real 1521773273]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773285 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.000609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.011402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.011520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.011522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.011524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.011527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.011544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.011554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.022283:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.022396:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.022398:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.022401:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.022403:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.022421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.022438:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.032462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.032579:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.032581:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.032583:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.032586:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.032603:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.032612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.043384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.043514:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.043531:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.043534:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.043536:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.043553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.043562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.054436:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.054537:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.054539:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.054542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.054544:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.054565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.054575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.065379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.065488:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.065490:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.065493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.065496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.065515:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.065539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.076409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.076545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.076547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.076552:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.076554:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.076572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.076588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.087592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.087715:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.087717:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.087720:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.087722:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.087739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.087748:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.097603:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.097722:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.097724:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.097727:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.097729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.097745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.097754:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.108214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.108335:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.108337:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.108340:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.108343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.108365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.108375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.117785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.117883:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.117885:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.117903:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.117906:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.117924:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.117933:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.127930:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.128046:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.128048:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.128051:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.128053:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.128071:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.128086:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.138095:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.138223:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.138225:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.138227:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.138230:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.138251:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.138261:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.148295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.148399:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.148401:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.148404:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.148406:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.148429:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.148439:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.158424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.158563:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.158567:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.158576:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.158581:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.158611:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.158622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.168419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.168521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.168523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.168526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.168528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.168542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.168552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.178612:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.178720:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.178722:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.178724:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.178726:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.178752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.178767:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.188422:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.188553:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.188557:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.188563:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.188568:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.188595:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.188606:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.198408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.198524:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.198526:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.198529:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.198531:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.198554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.198564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.208421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.208576:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.208578:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.208581:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.208584:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.208608:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.208620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.222251:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.222381:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.222383:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.222386:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.222388:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.222408:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.222419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.232702:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.232812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.232814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.232817:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.232819:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.232835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.232850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.241437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.241562:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.241564:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.241568:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.241570:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.241593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.241605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.251475:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.251582:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.251584:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.251587:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.251589:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.251602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.251612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.261393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.261505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.261515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.261533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26e9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.261535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.261550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.261560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.270807:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.270912:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.270914:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.270916:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ea1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.270918:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.270934:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.270944:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.280960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.281075:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.281077:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.281079:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ea3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.281082:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.281098:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.281113:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.290938:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.291040:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.291042:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.291044:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ea5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.291047:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.291130:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.291141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.300970:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.301073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.301074:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.301077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ea7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.301079:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.301094:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.301103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.311284:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.311400:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.311402:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.311405:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ea9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.311407:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.311424:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.311435:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.321400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.321537:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.321539:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.321544:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.321546:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.321565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.321575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.331439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.331542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.331544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.331546:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ead sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.331549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.331562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.331577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.341295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.341421:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.341423:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.341441:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eaf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.341444:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.341456:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.341465:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.351382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.351491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.351493:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.351496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eb1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.351498:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.351538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.351547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.361536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.361648:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.361650:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.361653:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eb3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.361655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.361673:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.361683:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.371394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.371503:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.371505:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.371516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eb5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.371518:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.371552:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.371562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.381392:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.381498:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.381500:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.381503:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eb7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.381513:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.381528:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.381543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.391339:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.391464:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.391466:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.391469:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eb9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.391471:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.391487:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.391496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.401943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.402053:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.402055:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.402059:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ebb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.402061:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.402077:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.402086:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.411621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.411723:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.411725:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.411728:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ebd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.411730:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.411740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.411750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.421607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.421711:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.421713:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.421716:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ebf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.421718:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.421735:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.421745:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.431471:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.431580:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.431582:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.431585:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ec1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.431587:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.431599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.431614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.441691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.441796:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.441798:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.441816:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ec3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.441818:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.441835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.441844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.452083:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.452187:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.452189:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.452192:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ec5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.452194:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.452209:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.452218:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.462589:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.462697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.462699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.462702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ec7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.462704:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.462720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.462730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.472491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.472615:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.472617:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.472619:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ec9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.472622:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.472632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.472642:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.482373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.482484:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.482486:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.482490:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ecb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.482492:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.482518:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.482534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.492440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.492551:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.492553:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.492556:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ecd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.492558:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.492577:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.492587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.502611:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.502711:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.502713:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.502715:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ecf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.502718:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.502734:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.502744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.513010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.513119:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.513121:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.513124:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ed1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.513126:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.513144:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.513169:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.524295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.524403:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.524405:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.524408:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ed3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.524410:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.524421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.524431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.535443:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.535581:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.535582:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.535585:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ed5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.535588:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.535610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.535627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.585455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.585579:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.585581:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.585584:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ed7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.585586:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.585604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.585614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.595429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.595524:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.595526:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.595528:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ed9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.595531:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.595543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.595553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.605535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.605643:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.605645:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.605650:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26edb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.605652:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.605673:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.605683:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.615221:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.615332:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.615334:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.615337:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26edd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.615339:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.615396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.615407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.625167:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.625292:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.625294:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.625297:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26edf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.625299:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.625318:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.625333:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.635391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.635499:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.635501:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.635504:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ee1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.635516:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.635541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.635551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.645401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.645519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.645521:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.645524:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ee3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.645526:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.645548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.645558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.652655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.652925:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.652927:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.652929:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ee5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.652931:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.652957:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.652968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.662754:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.662889:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.662894:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.662906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ee7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.662911:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.662934:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.662943:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.680011:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.680387:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.680389:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.680392:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ee9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.680394:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.680416:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.680431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.690442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.690552:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.690554:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.690558:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eeb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.690560:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.690586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.690596:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.700439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.700546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.700548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.700551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.700553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.700579:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.700590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.710937:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.711049:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.711051:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.711054:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.711056:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.711066:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.711075:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.721444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.721560:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.721562:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.721564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ef1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.721567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.721585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.721595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.732590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.732698:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.732701:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.732703:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ef3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.732706:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.732723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.732738:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.747422:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.747545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.747547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.747550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ef5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.747552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.747569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.747579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.758785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.758900:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.758902:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.758905:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ef7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.758907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.758922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.758931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.769451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.769566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.769568:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.769571:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ef9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.769573:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.769592:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.769602:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.779607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.779752:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.779754:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.779758:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26efb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.779760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.779798:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.779808:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.789857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.789973:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.789975:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.789978:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26efd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.789980:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.789996:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.790010:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.800206:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.800309:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.800311:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.800314:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26eff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.800316:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.800331:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.800341:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.808428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.808546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.808548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.808550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f01 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.808553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.808570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.808579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.818791:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.818902:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.818904:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.818907:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f03 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.818909:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.818925:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.818934:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.828915:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.829030:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.829032:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.829035:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f05 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.829037:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.829053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.829063:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.839043:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.839163:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.839165:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.839167:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f07 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.839170:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.839185:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.839199:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.849436:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.849556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.849558:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.849561:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f09 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.849563:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.849581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.849591:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.861188:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.861314:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.861316:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.861320:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f0b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.861323:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.861338:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.861358:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.871465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.871572:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.871574:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.871577:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f0d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.871579:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.871597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.871606:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.881494:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.881619:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.881621:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.881624:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f0f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.881626:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.881642:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.881652:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.893129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.893238:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.893240:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.893243:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f11 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.893261:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.893277:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.893291:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.903416:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.903527:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.903529:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.903532:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f13 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.903534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.903551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.903561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.915312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.915417:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.915419:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.915421:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f15 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.915424:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.915438:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.915447:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.926629:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.926768:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.926770:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.926773:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f17 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.926775:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.926806:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.926815:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.936911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.937025:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.937027:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.937029:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f19 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.937032:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.937048:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.937057:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.947066:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.947164:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.947181:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.947185:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f1b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.947187:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.947204:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.947219:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.957105:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.957226:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.957228:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.957231:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f1d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.957233:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.957249:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.957258:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.967401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.967519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.967521:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.967524:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f1f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.967526:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.967541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.967551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.979607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.979706:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.979708:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.979711:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f21 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.979713:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.979731:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.979740:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773274.990708:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773274.990813:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773274.990830:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773274.990833:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f23 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773274.990835:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773274.990852:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773274/real 1521773274]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773286 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773274.990862:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.001755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.001896:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.001898:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.001901:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f25 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.001903:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.001920:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.001935:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.012157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.012267:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.012269:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.012272:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f27 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.012274:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.012289:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.012299:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.022539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.022655:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.022657:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.022660:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f29 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.022662:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.022677:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.022687:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.032735:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.032848:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.032850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.032854:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f2b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.032856:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.032872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.032881:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.043286:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.043405:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.043407:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.043409:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f2d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.043412:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.043430:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.043442:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.054454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.054565:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.054567:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.054569:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f2f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.054572:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.054591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.054607:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.065859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.065970:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.065972:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.065975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f31 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.065977:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.065994:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.066004:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.076898:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.077027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.077029:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.077031:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f33 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.077034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.077054:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.077064:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.087225:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.087965:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.087967:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.087970:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f35 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.087972:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.088003:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.088015:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.098483:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.098620:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.098624:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.098630:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f37 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.098635:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.098677:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.098688:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.108611:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.108737:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.108739:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.108742:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f39 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.108744:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.108784:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.108815:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.117437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.117556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.117560:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.117569:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f3b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.117587:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.117611:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.117622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.126527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.126647:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.126649:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.126652:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f3d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.126654:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.126676:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.126686:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.136423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.136552:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.136554:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.136557:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f3f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.136559:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.136577:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.136587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.145615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.145719:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.145721:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.145724:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f41 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.145726:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.145741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.145751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.155481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.155586:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.155588:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.155591:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f43 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.155593:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.155606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.155621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.164564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.164662:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.164664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.164667:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f45 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.164669:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.164682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.164691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.174592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.174689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.174691:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.174694:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f47 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.174696:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.174717:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.174726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.184454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.184555:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.184557:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.184560:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f49 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.184562:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.184575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.184585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.194394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.194494:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.194496:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.194500:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f4b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.194503:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.194530:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.194540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.203886:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.203999:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.204001:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.204003:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f4d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.204006:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.204023:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.204038:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.209402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.211054:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.211056:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.211058:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f4f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.211060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.211108:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.211119:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.220303:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.220438:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.220440:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.220443:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f51 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.220445:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.220465:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.220475:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.229744:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.229840:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.229842:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.229844:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f53 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.229847:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.229865:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.229875:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.238977:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.239088:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.239090:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.239093:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f55 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.239095:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.239114:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.239123:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.247928:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.248035:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.248037:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.248040:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f57 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.248042:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.248060:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.248074:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.257894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.258003:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.258005:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.258008:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f59 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.258010:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.258027:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.258036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.267397:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.267492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.267494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.267498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f5b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.267500:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.267524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.267534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.276921:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.277015:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.277017:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.277020:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f5d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.277023:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.277040:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.277050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.285984:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.286081:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.286083:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.286086:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f5f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.286088:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.286101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.286110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.294911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.295073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.295075:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.295078:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f61 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.295080:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.295105:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.295119:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.303633:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.303732:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.303734:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.303737:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f63 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.303740:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.303752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.303761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.313434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.313543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.313545:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.313548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f65 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.313550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.313573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.313583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.323145:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.323246:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.323248:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.323251:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f67 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.323253:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.323271:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.323281:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.332567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.332670:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.332672:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.332675:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f69 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.332677:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.332696:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.332706:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.341529:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.341631:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.341633:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.341637:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f6b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.341640:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.341652:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.341667:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.350859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.350961:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.350963:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.350966:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f6d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.350968:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.350996:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.351005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.360445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.360554:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.360556:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.360559:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f6f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.360561:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.360583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.360593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.370230:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.370542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.370543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.370546:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f71 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.370549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.370570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.370582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.380268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.380385:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.380387:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.380389:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f73 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.380392:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.380412:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.380424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.390438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.390536:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.390538:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.390541:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f75 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.390543:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.390556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.390571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.400326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.400449:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.400451:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.400454:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f77 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.400456:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.400473:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.400482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.410434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.410534:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.410536:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.410539:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f79 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.410541:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.410553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.410563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.420446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.420555:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.420557:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.420561:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f7b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.420563:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.420590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.420600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.429898:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.429999:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.430001:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.430004:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f7d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.430006:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.430023:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.430032:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.438738:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.438851:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.438853:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.438856:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f7f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.438858:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.438877:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.438892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.447455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.447568:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.447570:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.447572:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f81 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.447575:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.447596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.447607:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.456486:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.456646:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.456648:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.456650:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f83 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.456653:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.456668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.456678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.466600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.466697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.466699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.466702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f85 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.466704:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.466723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.466733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.475465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.475576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.475578:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.475581:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f87 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.475583:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.475596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.475606:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.484800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.484900:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.484902:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.484905:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f89 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.484907:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.484924:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.484939:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.493662:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.493752:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.493754:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.493758:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f8b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.493761:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.493780:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.493824:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.503626:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.503729:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.503731:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.503734:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f8d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.503736:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.503753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.503763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.513532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.513630:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.513632:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.513635:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f8f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.513637:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.513649:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.513659:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.522653:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.522765:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.522767:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.522770:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f91 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.522772:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.522807:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.522816:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.532899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.533010:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.533012:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.533015:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f93 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.533017:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.533035:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.533051:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.541756:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.541866:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.541868:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.541871:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f95 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.541873:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.541889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.541898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.551697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.551803:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.551805:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.551808:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f97 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.551810:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.551828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.551838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.561665:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.561762:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.561764:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.561782:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f99 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.561784:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.561818:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.561827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.571498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.571622:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.571625:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.571629:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f9b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.571631:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.571643:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.571653:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.581459:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.581564:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.581566:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.581569:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f9d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.581571:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.581599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.581615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.591483:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.591598:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.591600:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.591603:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26f9f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.591605:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.591619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.591629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.601573:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.601685:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.601687:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.601690:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fa1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.601692:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.601705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.601714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.610904:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.611025:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.611026:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.611029:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fa3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.611031:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.611049:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.611058:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.619760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.619882:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.619884:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.619886:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fa5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.619888:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.619905:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.619914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.629613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.629722:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.629724:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.629727:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fa7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.629729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.629748:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.629763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.639379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.639482:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.639484:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.639487:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fa9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.639489:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.639534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.639544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.648802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.648918:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.648920:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.648924:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.648926:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.648943:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.648953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.658502:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.658607:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.658609:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.658611:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.658614:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.658626:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.658636:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.667399:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.667505:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.667517:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.667520:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26faf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.667522:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.667542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.667552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.676326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.676452:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.676454:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.676457:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fb1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.676459:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.676474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.676489:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.685200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.685304:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.685306:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.685309:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fb3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.685311:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.685325:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.685334:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.694500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.694607:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.694609:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.694611:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fb5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.694614:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.694625:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.694635:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.704016:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.704128:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.704130:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.704133:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fb7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.704135:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.704151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.704160:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.713920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.714020:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.714022:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.714025:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fb9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.714027:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.714053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.714063:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.722394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.722496:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.722498:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.722502:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fbb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.722505:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.722531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.722547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.731386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.731495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.731497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.731500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fbd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.731502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.731542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.731552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.740384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.740485:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.740487:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.740489:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fbf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.740492:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.740535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.740544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.749887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.749997:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.749999:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.750002:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fc1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.750004:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.750019:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.750029:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.759991:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.760113:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.760115:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.760117:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fc3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.760120:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.760135:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.760144:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.770391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.770489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.770491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.770494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fc5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.770497:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.770516:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.770531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.778934:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.779038:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.779040:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.779042:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fc7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.779045:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.779061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.779070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.787363:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.787473:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.787475:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.787477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fc9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.787480:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.787495:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.787505:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.796838:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.796938:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.796940:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.796944:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fcb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.796947:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.796963:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.796972:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.805966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.806076:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.806078:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.806080:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fcd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.806083:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.806098:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.806107:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.815095:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.815195:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.815197:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.815200:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fcf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.815202:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.815218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.815232:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.824013:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.824130:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.824132:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.824135:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fd1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.824137:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.824151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.824160:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.832736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.832834:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.832836:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.832839:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fd3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.832841:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.832867:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.832876:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.841430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.841521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.841523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.841526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fd5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.841528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.841540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.841549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.850861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.850976:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.850978:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.850981:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fd7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.850983:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.850999:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.851008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.861239:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.861380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.861382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.861385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fd9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.861387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.861408:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.861424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.870052:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.870165:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.870167:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.870171:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fdb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.870173:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.870188:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.870197:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.879858:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.879957:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.879959:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.879962:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fdd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.879964:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.879982:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.879991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.888397:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.888514:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.888516:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.888519:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fdf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.888521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.888542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.888551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.898424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.898545:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.898547:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.898550:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fe1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.898552:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.898578:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.898589:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.908525:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.908629:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.908631:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.908634:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fe3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.908637:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.908649:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.908664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.917874:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.917977:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.917980:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.917982:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fe5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.917985:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.918004:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.918013:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.927312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.927421:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.927423:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.927426:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fe7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.927428:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.927448:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.927459:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.936863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.936959:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.936961:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.936964:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fe9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.936966:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.936984:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.936993:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.945802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.945900:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.945902:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.945906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26feb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.945908:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.945920:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.945930:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.955500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.955610:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.955612:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.955614:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.955617:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.955629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.955644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.964906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.965002:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.965004:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.965007:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.965010:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.965022:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.965031:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.973467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.973566:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.973568:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.973570:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ff1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.973573:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.973585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.973595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.982426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.982520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.982522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.982525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ff3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.982527:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.982541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.982551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773275.991531:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773275.991633:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773275.991635:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773275.991638:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ff5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773275.991641:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773275.991653:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773275.991662:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.000187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.000304:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.000306:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.000309:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ff7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.000311:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.000328:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773275/real 1521773275]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773287 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.000343:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.009498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.009629:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.009631:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.009634:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ff9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.009637:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.009656:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.009666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.019423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.019535:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.019537:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.019541:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ffb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.019543:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.019569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.019580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.029407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.029514:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.029517:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.029519:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26ffd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.029522:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.029550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.029560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.038530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.038628:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.038630:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.038632:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x26fff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.038635:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.038647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.038656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.047963:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.048058:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.048060:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.048063:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27001 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.048065:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.048077:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.048091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.055469:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0F:1521773276.055637:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.055640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.055645:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27003 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.055648:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.055669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.055679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.064467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.064581:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.064584:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.064588:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27005 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.064591:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.064611:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.064621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.073474:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.073572:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.073577:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.073580:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27007 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.073582:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.073594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.073604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.082279:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.082393:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.082395:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.082398:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27009 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.082401:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.082412:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.082422:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.091062:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.091171:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.091176:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.091179:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2700b sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.091181:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.091194:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.091209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.100407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.100689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.100691:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.100698:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2700d sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.100700:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.100721:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.100731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.110406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.110625:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.110629:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.110632:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2700f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.110635:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.110648:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.110657:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.119255:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.119360:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.119362:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.119365:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27011 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.119367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.119381:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.119391:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.128295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.128392:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.128394:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.128397:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27013 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.128400:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.128421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.128431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.138049:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.138146:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.138149:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.138151:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27015 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.138156:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.138172:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.138188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.147611:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.147706:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.147710:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.147714:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27017 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.147717:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.147727:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.147736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.157622:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.157712:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.157714:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.157717:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27019 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.157720:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.157739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.157749:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.167460:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.167583:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.167585:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.167588:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2701b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.167591:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.167614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.167624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.176540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.176684:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.176686:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.176689:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2701d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.176691:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.176706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.176716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.185746:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.185965:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.185970:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.185973:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2701f sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.185975:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.185995:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.186011:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.194387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.194950:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.194955:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.194957:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27021 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.194961:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.195016:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.195028:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.203600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.203704:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.203706:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.203709:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27023 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.203712:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.203731:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.203741:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.213165:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.213268:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.213271:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.213276:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27025 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.213278:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.213296:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.213305:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.222428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.222556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.222558:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.222561:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27027 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.222564:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.222585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.222595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.232485:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.232607:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.232609:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.232616:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27029 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.232618:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.232644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.232660:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.241971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.242075:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.242080:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.242083:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2702b sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.242085:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.242097:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.242106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.251370:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.251464:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.251468:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.251471:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2702d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.251474:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.251491:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.251501:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.261305:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.261616:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.261619:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.261622:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2702f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.261625:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.261647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.261657:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.271252:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.271359:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.271361:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.271364:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27031 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.271367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.271387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.271397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.280898:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.280999:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.281002:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.281004:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27033 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.281007:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.281026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.281041:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.289846:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.289938:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.289942:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.289945:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27035 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.289947:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.289959:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.289968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.299517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.299676:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.299678:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.299681:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27037 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.299684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.299744:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.299755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.309423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.309541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.309543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.309546:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27039 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.309549:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.309569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.309580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.319102:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.319212:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.319215:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.319218:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2703b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.319221:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.319240:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.319249:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.329146:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.329255:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.329257:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.329260:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2703d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.329262:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.329281:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.329296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.338136:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.338359:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.338362:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.338364:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2703f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.338367:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.338387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.338397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.347721:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.347825:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.347827:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.347830:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27041 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.347833:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.347853:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.347862:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.356452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.356554:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.356556:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.356559:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27043 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.356562:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.356585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.356596:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.366292:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.366415:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.366417:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.366420:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27045 sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.366423:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.366434:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.366444:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.375674:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.375787:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.375789:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.375792:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27047 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.375795:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.375814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.375829:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.385941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.386052:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.386054:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.386058:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27049 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.386061:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.386081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.386091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.395025:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.395122:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.395124:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.395127:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2704b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.395129:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.395148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.395158:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.404402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.404511:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.404514:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:1.0:1521773276.404516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2704d sending 1 waiting 0: failed 5
00000800:00000100:1.0:1521773276.404519:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.404541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.404552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.414127:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.414237:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.414239:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.414242:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2704f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.414245:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.414257:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.414266:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.423302:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.423417:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.423419:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.423423:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27051 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.423426:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.423440:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.423455:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.432129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.432232:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.432235:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.432238:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27053 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.432241:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.432260:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.432269:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.442454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:1.0:1521773276.442563:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:1.0:1521773276.442566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.442569:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27055 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.442573:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.442595:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.442605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.451478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.451591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.451593:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.451596:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27057 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.451598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.451619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.451629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.460586:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.460679:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.460681:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.460684:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27059 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.460687:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.460698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.460708:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.469788:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.469892:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.469894:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.469897:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2705b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.469899:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.469911:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.469926:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.478425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.478525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.478527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.478530:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2705d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.478532:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.478554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.478564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.487385:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.487491:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.487493:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.487495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2705f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.487498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.487519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.487529:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.496412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.496516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.496518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.496520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27061 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.496522:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.496546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.496556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.505874:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.505980:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.505982:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.505984:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27063 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.505986:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.505998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.506007:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.515418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.515525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.515527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.515530:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27065 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.515533:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.515553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.515568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.525331:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.525454:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.525456:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.525459:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27067 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.525461:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.525479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.525489:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.534894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.534994:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.534996:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.534999:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27069 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.535001:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.535014:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.535023:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.544500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.544618:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.544620:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.544623:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2706b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.544625:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.544638:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.544648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.554400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.554516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.554518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.554521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2706d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.554524:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.554543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.554553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.564174:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.564278:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.564280:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.564283:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2706f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.564285:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.564303:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.564317:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.572994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.573095:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.573097:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.573100:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27071 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.573102:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.573120:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.573130:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.582453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.582572:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.582574:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.582577:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27073 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.582579:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.582599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.582609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.591916:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.592024:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.592026:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.592029:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27075 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.592032:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.592045:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.592054:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.601096:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.601193:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.601195:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.601197:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27077 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.601200:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.601218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.601227:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.611063:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.611167:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.611169:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.611172:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27079 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.611174:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.611194:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.611209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.620797:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.620903:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.620905:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.620908:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2707b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.620911:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.620922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.620931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.630182:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.630289:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.630291:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.630294:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2707d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.630296:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.630313:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.630323:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.639268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.639370:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.639372:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.639375:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2707f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.639378:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.639391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.639400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.649023:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.649130:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.649132:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.649135:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27081 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.649138:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.649157:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.649166:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.658605:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.658709:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.658711:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.658714:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27083 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.658717:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.658735:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.658751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.668396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.668502:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.668504:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.668517:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27085 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.668521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.668577:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.668589:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.677330:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.677457:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.677459:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.677462:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27087 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.677464:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.677479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.677488:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.686377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.686491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.686493:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.686496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27089 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.686498:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.686536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.686546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.695085:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.695195:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.695197:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.695200:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2708b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.695202:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.695219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.695228:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.703831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.703954:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.703956:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.703959:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2708d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.703961:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.703977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.703991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.712482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.712602:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.712604:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.712607:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2708f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.712609:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.712620:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.712630:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.721929:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.722031:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.722034:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.722036:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27091 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.722039:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.722073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.722083:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.730888:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.730992:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.730994:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.730997:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27093 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.730999:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.731014:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.731024:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.740004:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.740144:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.740146:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.740149:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27095 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.740152:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.740174:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.740185:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.748900:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.749015:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.749017:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.749020:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27097 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.749022:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.749037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.749051:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.757542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.757654:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.757656:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.757659:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27099 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.757661:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.757672:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.757682:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.766319:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.766425:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.766427:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.766429:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2709b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.766432:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.766441:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.766451:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.774998:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.775106:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.775108:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.775111:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2709d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.775113:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.775124:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.775134:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.784288:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.784393:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.784395:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.784398:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2709f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.784400:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.784412:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.784422:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.793136:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.793246:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.793248:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.793251:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270a1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.793253:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.793269:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.793304:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.802979:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.803097:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.803099:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.803101:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270a3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.803104:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.803120:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.803129:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.812565:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.812677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.812680:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.812682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270a5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.812686:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.812703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.812713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.822263:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.822392:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.822394:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.822397:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270a7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.822400:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.822415:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.822427:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.831103:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.831212:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.831214:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.831217:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270a9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.831219:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.831231:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.831240:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.839974:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.840091:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.840093:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.840096:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270ab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.840098:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.840113:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.840128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.849849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.849962:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.849964:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.849967:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270ad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.849969:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.849985:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.849994:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.859411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.859534:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.859536:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.859539:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270af sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.859542:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.859558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.859568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.869180:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.869285:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.869287:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.869305:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270b1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.869307:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.869337:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.869376:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.879239:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.879343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.879354:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.879377:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270b3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.879379:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.879395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.879406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.888372:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.888473:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.888475:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.888478:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270b5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.888482:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.888497:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.888521:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.895970:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.896102:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.896104:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.896106:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270b7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.896109:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.896125:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.896135:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.905421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.905532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.905534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.905537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270b9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.905539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.905560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.905569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.914802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.914922:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.914924:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.914927:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270bb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.914929:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.914939:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.914948:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.923391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.923501:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.923503:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.923515:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270bd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.923518:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.923535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.923545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.933047:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.933147:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.933149:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.933152:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270bf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.933154:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.933164:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.933194:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.942080:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.942206:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.942208:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.942211:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270c1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.942213:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.942229:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.942239:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.951270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.951375:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.951377:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.951379:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270c3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.951382:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.951393:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.951403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.960222:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.960340:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.960342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.960353:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270c5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.960372:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.960409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.960421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.969933:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.970039:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.970041:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.970044:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270c7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.970046:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.970061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.970070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.979877:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.979980:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.979982:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.979985:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270c9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.979987:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.980001:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.980016:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.989381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.989483:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.989485:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.989487:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270cb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.989490:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.989515:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.989525:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773276.998996:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773276.999108:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773276.999110:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773276.999113:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270cd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773276.999115:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773276.999132:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773276/real 1521773276]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773288 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773276.999141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.007923:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.008022:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.008024:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.008027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270cf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.008029:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.008045:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.008055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.016979:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.017089:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.017091:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.017093:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270d1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.017096:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.017111:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.017120:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.025732:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.025850:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.025852:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.025855:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270d3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.025857:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.025873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.025887:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.034428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.034527:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.034529:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.034531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270d5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.034535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.034547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.034556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.043279:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.043380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.043382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.043385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270d7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.043387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.043405:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.043417:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.052526:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.052629:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.052631:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.052634:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270d9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.052636:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.052647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.052656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.062388:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.062498:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.062500:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.062503:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270db sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.062505:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.062529:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.062538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.072518:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.072625:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.072627:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.072630:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270dd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.072632:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.072651:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.072667:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.082432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.082530:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.082532:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.082535:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270df sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.082537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.082550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.082560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.091723:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.091835:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.091837:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.091855:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270e1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.091857:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.091889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.091898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.101016:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.101155:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.101159:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.101165:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270e3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.101170:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.101196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.101206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.110395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.110518:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.110520:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.110522:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270e5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.110534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.110557:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.110568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.119418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.119553:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.119557:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.119563:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270e7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.119567:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.119597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.119614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.128068:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.128218:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.128220:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.128222:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270e9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.128224:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.128245:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.128255:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.136647:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.136743:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.136745:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.136748:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270eb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.136750:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.136787:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.136813:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.145927:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.146065:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.146067:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.146070:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270ed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.146072:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.146093:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.146103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.154811:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.154951:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.154953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.154955:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270ef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.154958:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.154979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.154988:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.163671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.163807:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.163808:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.163811:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.163816:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.163838:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.163852:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.172594:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.172697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.172699:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.172701:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.172704:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.172716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.172726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.182602:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.182708:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.182710:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.182713:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.182716:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.182738:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.182748:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.191562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.191665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.191667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.191670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.191672:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.191684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.191694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.201545:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.201651:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.201653:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.201655:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.201657:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.201680:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.201690:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.210260:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.210397:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.210399:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.210402:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.210404:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.210416:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.210430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.219157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.219255:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.219257:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.219260:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.219262:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.219296:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.219305:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.228894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.229003:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.229005:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.229007:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x270ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.229009:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.229026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.229035:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.237627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.237726:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.237728:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.237731:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27101 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.237733:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.237749:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.237759:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.246859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.246936:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.246940:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27103 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773277.246942:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.246958:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.246967:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.255876:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.255993:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.255995:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.255998:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27105 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.256000:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.256011:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.256026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.265442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.265545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.265547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.265550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27107 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.265552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.265564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.265574:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.275421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.275544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.275546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.275549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27109 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.275551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.275573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.275583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.284421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.284549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.284553:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2710b sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773277.284555:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.284579:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.284590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.294427:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.294556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.294560:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.294566:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2710d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.294571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.294599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.294609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.304562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.304668:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.304670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.304673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2710f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.304675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.304687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.304702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.313873:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.313981:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.313983:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.313986:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27111 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.313988:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.313999:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.314008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.323821:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.323935:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.323936:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.323939:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27113 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.323941:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.323957:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.323967:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.332617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.332696:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.332700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27115 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773277.332702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.332718:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.332728:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.342606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.342715:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.342717:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.342721:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27117 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.342723:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.342741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.342750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.351955:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.352057:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.352059:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.352062:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27119 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.352064:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.352081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.352095:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.360668:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.360788:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.360790:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.360793:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2711b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.360795:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.360806:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.360815:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.370614:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.370718:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.370720:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.370723:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2711d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.370725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.370742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.370752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.379581:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.379691:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.379693:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.379696:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2711f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.379698:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.379715:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.379725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.389411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.389519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.389521:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.389524:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27121 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.389526:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.389544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.389554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.399470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.399578:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.399580:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.399583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27123 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.399585:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.399597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.399612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.408782:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.408905:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.408907:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.408910:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27125 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.408912:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.408929:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.408938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.417851:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.417956:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.417958:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.417961:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27127 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.417963:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.417978:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.417987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.426827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.426954:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.426956:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.426959:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27129 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.426961:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.426976:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.426985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.437286:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.437401:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.437403:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.437406:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2712b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.437408:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.437424:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.437436:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.446060:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.446182:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.446184:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.446187:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2712d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.446189:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.446205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.446220:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.455734:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.455850:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.455853:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.455855:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2712f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.455858:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.455869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.455878:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.465820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.465925:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.465926:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.465929:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27131 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.465931:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.465961:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.465969:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.474603:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.474706:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.474708:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.474711:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27133 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.474714:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.474730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.474740:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.483268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.483372:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.483374:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.483377:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27135 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.483379:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.483389:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.483398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.492010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.492125:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.492127:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.492130:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27137 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.492133:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.492148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.492163:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.501793:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.501893:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.501895:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.501898:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27139 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.501900:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.501916:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.501925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.510504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.510625:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.510627:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.510630:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2713b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.510632:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.510643:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.510653:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.519651:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.519763:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.519765:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.519768:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2713d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.519786:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.519797:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.519806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.528658:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.528756:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.528758:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.528761:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2713f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.528764:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.528795:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.528804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.538305:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.538440:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.538442:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.538445:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27141 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.538447:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.538464:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.538479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.548322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.548433:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.548435:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.548438:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27143 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.548440:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.548456:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.548465:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.557911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.558030:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.558032:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.558034:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27145 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.558036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.558052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.558061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.567971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.568079:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.568080:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.568084:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27147 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.568086:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.568101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.568110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.577438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.577548:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.577550:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.577553:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27149 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.577555:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.577574:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.577585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.586241:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.586342:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.586344:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.586358:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2714b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.586375:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.586399:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.586417:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.596166:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.596270:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.596272:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.596275:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2714d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.596277:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.596308:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.596318:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.605445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.605569:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.605571:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.605574:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2714f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.605576:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.605595:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.605605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.615872:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.615981:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.615983:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.615986:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27151 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.615988:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.615999:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.616008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.625569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.625681:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.625683:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.625686:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27153 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.625689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.625705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.625714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.635374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.635476:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.635478:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.635480:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27155 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.635483:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.635499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.635539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.645241:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.645368:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.645370:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.645374:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27157 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.645376:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.645395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.645407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.654825:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.654932:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.654934:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.654937:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27159 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.654939:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.654956:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.654966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.664833:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.664957:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.664959:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.664961:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2715b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.664964:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.664980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.664990:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.674404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.674516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.674518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.674521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2715d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.674523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.674541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.674552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.684373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.684488:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.684490:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.684493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2715f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.684496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.684519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.684534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.693318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.693424:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.693426:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.693429:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27161 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.693432:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.693472:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.693482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.702442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.702574:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.702578:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.702583:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27163 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.702588:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.702615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.702626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.711619:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.711743:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.711745:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.711748:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27165 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.711750:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.711761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.711770:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.720599:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.720705:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.720707:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.720711:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27167 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.720714:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.720735:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.720744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773277.729539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.729638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.729640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d095d8) failed: 5
00000800:00000100:3.0:1521773277.729642:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27169 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.729645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.729669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.729685:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.738432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.738550:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.738551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.738554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2716c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.738556:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.738586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.738597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.747299:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.747450:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.747452:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.747454:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2716e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.747457:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.747476:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.747486:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.757232:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.757379:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.757381:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.757383:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27170 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.757385:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.757407:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.757419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.763455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.764492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.764494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.764496:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27172 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.764498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.764569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.764581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.774053:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.774191:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.774193:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.774195:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27174 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.774199:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.774214:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.774224:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.784722:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.784841:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.784843:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.784845:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27176 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.784847:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.784863:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.784873:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.793700:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.793828:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.793830:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.793832:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27178 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.793834:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.793854:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.793865:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.802782:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.802902:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.802903:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.802906:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2717a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.802908:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.802928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.802938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.813492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.813617:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.813618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.813621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2717c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.813623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.813639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.813649:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.822304:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.822432:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.822434:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.822436:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2717e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.822438:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.822458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.822470:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.831280:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.831410:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.831412:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.831415:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27180 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.831418:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.831442:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.831453:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.840915:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.841033:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.841034:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.841036:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27182 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.841039:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.841055:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.841065:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.850106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.850229:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.850231:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.850233:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27184 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.850236:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.850255:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.850270:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.859736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.859866:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.859867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.859870:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27186 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.859872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.859887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.859896:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.869086:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.869203:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.869204:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.869222:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27188 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.869224:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.869244:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.869254:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.879585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.879738:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.879740:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.879742:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2718a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.879744:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.879762:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.879772:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.888322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.888472:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.888475:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.888477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2718c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.888479:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.888499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.888533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.897191:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.897312:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.897313:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.897316:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2718e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.897318:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.897337:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.897379:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.906556:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.906683:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.906685:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.906688:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27190 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.906690:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.906707:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.906717:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.915557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.915677:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.915679:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.915682:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27192 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.915684:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.915705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.915716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.924529:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.924788:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.924789:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.924792:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27194 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.924794:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.924830:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.924840:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.933582:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.933699:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.933700:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.933703:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27196 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.933705:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.933726:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.933736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.942155:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.942473:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.942474:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.942477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27198 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.942479:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.942501:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.942542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.951911:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.952030:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.952031:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.952034:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2719a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.952036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.952053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.952063:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.961772:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.961898:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.961900:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.961902:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2719c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.961904:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.961921:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.961931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.970884:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.971012:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.971013:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.971016:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2719e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.971018:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.971039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.971055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.979428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.979553:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.979555:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.979557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.979559:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.979588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.979599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.988313:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.988467:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.988468:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.988471:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.988473:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.988494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.988504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773277.997136:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773277.997263:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773277.997265:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773277.997267:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773277.997270:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773277.997288:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773277/real 1521773277]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773289 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773277.997298:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.006003:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.006123:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.006125:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.006128:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.006130:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.006151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.006161:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.015502:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.015629:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.015631:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.015633:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.015635:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.015646:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.015656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.025287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.025432:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.025434:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.025436:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.025439:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.025456:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.025467:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.034773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.034908:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.034910:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.034912:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.034914:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.034931:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.034941:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.043552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.043674:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.043676:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.043678:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.043682:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.043703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.043713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.052430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.052538:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.052540:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.052542:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.052545:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.052561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.052571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.062379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.062502:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.062503:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.062511:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.062513:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.062547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.062558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.071362:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.071495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.071497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.071499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.071501:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.071540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.071551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.081158:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.081296:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.081298:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.081300:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.081302:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.081317:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.081327:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.090982:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.091098:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.091100:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.091102:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.091105:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.091122:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.091138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.099925:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.100086:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.100088:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.100092:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.100094:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.100119:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.100129:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.109124:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.109284:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.109288:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.109292:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.109297:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.109329:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.109339:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.119189:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.119328:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.119330:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.119332:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.119334:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.119361:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.119371:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.128149:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.128300:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.128304:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.128309:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.128314:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.128344:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.128379:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.138180:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.138304:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.138306:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.138318:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.138320:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.138340:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.138362:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.146952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.147120:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.147123:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.147128:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.147133:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.147166:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.147176:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.157134:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.157288:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.157294:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.157299:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.157304:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.157328:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.157338:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.167235:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.167363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.167365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.167367:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.167371:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.167386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.167397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.176333:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.176490:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.176491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.176494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.176496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.176538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.176548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.185503:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.185664:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.185666:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.185668:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.185670:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.185686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.185696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.194568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.194696:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.194698:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.194700:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.194703:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.194728:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.194739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.203482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.203652:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.203656:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.203661:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.203678:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.203692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.203701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.213240:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.213399:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.213400:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.213403:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.213405:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.213430:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.213448:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.222709:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.222843:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.222844:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.222847:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.222864:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.222882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.222892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.232313:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.232445:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.232446:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.232449:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.232451:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.232468:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.232479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.241188:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.241305:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.241307:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.241309:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.241312:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.241344:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.241368:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.249989:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.250112:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.250114:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.250116:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.250118:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.250135:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.250146:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.259445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.259573:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.259574:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.259577:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.259579:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.259594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.259605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.268747:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.268886:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.268888:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.268890:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.268892:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.268911:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.268920:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.278387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.278501:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.278503:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.278515:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.278517:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.278537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.278548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.287391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.287495:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.287496:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.287499:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.287501:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.287524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.287534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.296552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.296670:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.296671:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.296674:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.296676:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.296695:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.296705:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.305396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.305544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.305546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.305548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.305551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.305570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.305580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.314399:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.314548:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.314550:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.314553:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.314556:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.314576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.314586:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.323861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.323974:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.323975:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.323978:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.323980:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.323997:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.324007:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.333298:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.333419:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.333421:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.333439:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.333441:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.333461:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.333476:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.341989:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.342125:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.342126:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.342129:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.342131:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.342148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.342158:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.351234:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.351342:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.351344:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.351357:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.351360:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.351377:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.351388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.360397:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.360655:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.360657:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.360660:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.360662:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.360681:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.360691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.370555:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.370673:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.370675:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.370679:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.370681:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.370700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.370711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.380966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.381096:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.381097:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.381100:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.381102:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.381119:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.381129:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.390416:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.390544:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.390545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.390548:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.390550:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.390570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.390580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.400858:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.400980:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.400982:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.400984:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.400986:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.400998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.401008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.409865:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.410002:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.410004:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.410006:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.410008:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.410025:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.410035:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.419572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.419683:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.419684:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.419687:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x271fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.419689:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.419708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.419718:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.429157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.429280:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.429283:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.429285:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27200 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.429287:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.429304:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.429314:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.439489:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.439617:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.439619:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.439621:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27202 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.439624:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.439636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.439646:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.449451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.449575:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.449577:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.449579:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27204 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.449582:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.449603:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.449614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.458906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.459042:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.459043:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.459046:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27206 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.459048:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.459065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.459080:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.468945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.469152:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.469154:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.469156:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27208 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.469158:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.469177:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.469186:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.478677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.478805:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.478806:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.478824:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2720a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.478827:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.478844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.478854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.488736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.488868:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.488870:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.488872:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2720c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.488874:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.488891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.488902:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.498889:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.499008:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.499010:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.499012:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2720e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.499014:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.499031:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.499041:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.508267:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.508395:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.508397:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.508400:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27210 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.508402:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.508426:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.508438:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.516811:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.516940:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.516942:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.516945:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27212 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.516947:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.516970:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.516982:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.525643:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.525767:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.525769:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.525771:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27214 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.525773:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.525791:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.525801:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.536097:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.536205:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.536207:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.536209:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27216 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.536212:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.536229:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.536238:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.545466:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.545582:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.545584:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.545586:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27218 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.545589:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.545604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.545615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.554865:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.554977:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.554979:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.554981:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2721a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.554984:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.555001:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.555010:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.564371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.564482:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.564483:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.564485:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2721c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.564488:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.564504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.564522:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.573407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.573520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.573522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.573525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2721e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.573527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.573546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.573557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.581793:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.581914:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.581916:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.581918:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27220 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.581920:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.581932:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.581946:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.591209:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.591312:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.591314:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.591316:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27222 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.591320:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.591339:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.591361:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.599748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.599865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.599866:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.599868:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27224 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.599871:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.599882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.599892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.607652:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.607773:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.607775:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.607777:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27226 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.607779:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.607797:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.607806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.616104:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.616209:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.616211:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.616213:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27228 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.616215:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.616226:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.616235:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.624593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.624697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.624699:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.624701:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2722a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.624703:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.624714:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.624722:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.633229:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.633339:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.633340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.633343:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2722c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.633347:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.633371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.633380:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.642559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.642677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.642678:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.642682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2722e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.642684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.642701:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.642710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.651020:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.651130:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.651132:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.651134:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27230 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.651136:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.651148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.651158:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.659495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.659618:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.659620:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.659623:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27232 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.659625:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.659640:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.659649:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.667850:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.667964:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.667966:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.667968:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27234 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.667970:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.667991:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.668002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.677406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.677527:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.677529:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.677531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27236 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.677533:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.677553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.677563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.687044:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.687164:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.687165:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.687168:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27238 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.687170:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.687180:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.687190:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.695952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.696063:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.696066:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.696068:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2723a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.696070:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.696086:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.696101:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.705048:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.705174:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.705176:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.705179:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2723c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.705181:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.705194:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.705204:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.713631:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.713752:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.713753:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.713756:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2723e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.713758:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.713774:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.713784:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.721964:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.722064:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.722065:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.722067:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27240 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.722070:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.722085:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.722094:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.731534:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.731645:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.731647:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.731649:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27242 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.731651:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.731662:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.731672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.740371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.740473:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.740475:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.740477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27244 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.740479:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.740489:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.740498:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.751524:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.751639:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.751640:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.751643:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27246 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.751645:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.751662:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.751672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.760061:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.760168:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.760170:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.760172:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27248 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.760174:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.760184:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.760194:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.768528:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.768648:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.768649:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.768652:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2724a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.768654:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.768670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.768680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.776937:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.777048:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.777050:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.777052:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2724c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.777054:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.777066:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.777075:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.785794:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.785909:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.785911:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.785913:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2724e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.785915:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.785925:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.785935:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.794913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.795081:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.795082:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.795085:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27250 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.795087:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.795103:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.795113:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.804583:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.804695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.804696:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.804699:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27252 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.804701:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.804716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.804726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.813179:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.813296:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.813298:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.813300:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27254 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.813302:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.813318:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.813333:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.822129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.822240:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.822242:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.822244:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27256 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.822246:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.822261:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.822272:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.830635:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.830742:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.830744:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.830746:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27258 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.830748:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.830763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.830772:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.845745:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.846139:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.846141:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.846143:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2725a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.846145:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.846185:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.846196:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.855823:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.855930:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.855931:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.855933:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2725c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.855937:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.855947:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.855957:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.865495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.865632:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.865633:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.865636:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2725e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.865638:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.865653:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.865664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.874276:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.874392:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.874394:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.874396:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27260 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.874398:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.874410:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.874421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.882996:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.883104:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.883105:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.883108:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27262 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.883110:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.883120:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.883129:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.891587:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.891697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.891699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.891701:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27264 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.891703:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.891713:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.891723:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.901386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.901497:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.901499:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.901501:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27266 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.901504:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.901530:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.901540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.910697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.910813:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.910814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.910818:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27268 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.910820:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.910835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.910844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.919425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.919539:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.919541:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.919543:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2726a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.919545:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.919565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.919576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.928642:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.928745:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.928746:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.928749:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2726c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.928751:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.928761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.928770:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.936974:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.937100:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.937102:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.937104:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2726e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.937106:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.937117:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.937131:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.945852:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.945963:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.945965:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.945967:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27270 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.945970:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.945985:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.945995:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.957124:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.957281:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.957282:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.957285:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27272 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.957287:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.957303:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.957314:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.965844:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.965951:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.965953:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.965956:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27274 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.965958:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.965973:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.965983:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.975190:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.975309:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.975311:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.975313:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27276 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.975315:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.975330:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.975341:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.984562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.984687:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.984689:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.984691:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27278 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.984693:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.984710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.984720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773278.993374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773278.993479:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773278.993481:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773278.993483:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2727a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773278.993485:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773278.993500:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773278/real 1521773278]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773290 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773278.993519:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.002027:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.002129:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.002130:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.002132:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2727c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.002135:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.002146:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.002156:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.010609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.010735:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.010736:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.010739:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2727e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.010741:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.010756:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.010767:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.019007:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.019122:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.019123:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.019125:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27280 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.019127:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.019138:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.019148:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.027591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.027718:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.027720:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.027722:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27282 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.027724:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.027740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.027750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.036216:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.036355:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.036357:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.036359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27284 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.036362:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.036382:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.036393:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.044737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.044855:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.044857:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.044859:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27286 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.044861:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.044872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.044882:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.053425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.053538:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.053539:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.053541:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27288 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.053543:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.053568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.053583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.062024:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.062160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.062162:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.062165:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2728a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.062168:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.062194:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.062206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.070795:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.070911:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.070912:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.070915:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2728c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.070917:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.070937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.070947:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.079531:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.079660:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.079661:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.079664:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2728e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.079666:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.079686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.079697:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.088030:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.088149:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.088150:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.088153:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27290 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.088155:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.088170:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.088180:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.097821:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.097926:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.097927:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.097929:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27292 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.097931:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.097942:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.097951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.106250:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.106368:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.106370:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.106373:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27294 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.106375:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.106391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.106400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.115277:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.115402:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.115404:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.115406:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27296 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.115410:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.115434:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.115444:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.123729:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.123849:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.123851:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.123853:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27298 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.123855:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.123879:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.123889:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.132552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.132680:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.132681:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.132684:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2729a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.132686:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.132700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.132709:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.141161:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.141282:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.141284:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.141286:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2729c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.141288:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.141309:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.141319:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.149770:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.149903:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.149904:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.149906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2729e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.149909:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.149929:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.149939:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.164865:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.164989:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.164991:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.164993:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.164995:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.165018:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.165028:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.173678:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.173800:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.173801:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.173805:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.173807:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.173820:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.173835:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.183327:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.183461:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.183463:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.183465:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.183467:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.183490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.183500:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.192742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.192866:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.192867:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.192870:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.192872:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.192895:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.192905:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.208341:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.208494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.208496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.208498:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.208500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.208528:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.208538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.216971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.217262:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.217264:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.217266:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.217269:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.217291:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.217303:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.226387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.226505:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.226513:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.226515:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.226517:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.226543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.226554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.235245:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.235367:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.235370:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.235372:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.235375:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.235395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.235405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.244661:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.244769:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.244771:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.244773:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.244775:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.244786:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.244796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.259434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.259580:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.259582:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.259585:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.259587:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.259642:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.259654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.268740:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.268858:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.268859:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.268862:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.268864:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.268881:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.268892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.277382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.277492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.277494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.277496:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.277498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.277517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.277527:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.287157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.287269:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.287271:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.287273:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.287275:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.287292:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.287302:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.295766:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.295882:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.295883:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.295885:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.295887:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.295906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.295916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.304446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.304564:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.304566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.304568:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.304570:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.304585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.304601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.312931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.313053:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.313055:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.313057:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.313060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.313073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.313082:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.321543:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.321664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.321666:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.321668:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.321671:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.321690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.321700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.330201:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.330329:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.330330:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.330333:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.330335:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.330364:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.330374:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.338820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.338931:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.338933:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.338935:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.338937:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.338955:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.338966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.347440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.347552:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.347554:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.347556:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.347559:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.347571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.347581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.360541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.360690:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.360691:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.360694:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.360696:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.360715:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.360725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.369343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.369461:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.369462:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.369465:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.369467:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.369484:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.369494:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.378281:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.378400:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.378402:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.378404:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.378406:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.378425:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.378436:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.387680:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.387797:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.387798:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.387801:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.387803:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.387815:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.387825:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.397162:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.397282:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.397283:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.397286:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.397289:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.397306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.397316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.405727:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.405841:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.405843:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.405845:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.405847:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.405859:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.405868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.415449:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.415575:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.415576:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09350) failed: 5
00000800:00000100:3.0:1521773279.415579:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x272d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.415581:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.415594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.415604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.425111:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773279.425235:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773279.425237:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773279.425240:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2733e sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773279.425242:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.425255:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.425269:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.433821:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.433944:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.433946:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.433948:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27340 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.433951:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.433969:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.433979:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.440964:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.441105:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.441107:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.441109:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27342 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.441111:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.441129:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.441139:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.450125:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.450247:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.450249:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.450251:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27344 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.450253:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.450271:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.450281:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.459760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.459875:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.459877:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.459879:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27346 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.459881:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.459903:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.459913:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.469543:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.469668:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.469669:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.469672:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27348 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.469674:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.469693:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.469703:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.478373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.478496:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.478497:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.478500:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2734a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.478502:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.478526:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.478536:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.487736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.487850:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.487852:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.487854:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2734c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.487856:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.487874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.487885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.497418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.497523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.497525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.497527:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2734e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.497529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.497541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.497551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.506092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.506213:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.506215:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.506217:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27350 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.506220:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.506237:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.506246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.515142:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.515257:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.515259:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.515261:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27352 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.515263:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.515281:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.515292:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.523851:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.523964:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.523966:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.523968:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27354 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.523970:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.523987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.523997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.533767:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.533882:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.533884:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.533886:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27356 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.533888:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.533906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.533916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.543273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.543390:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.543392:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.543394:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27358 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.543396:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.543410:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.543426:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.551702:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.551825:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.551826:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.551829:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2735a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.551831:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.551843:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.551852:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.560599:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.560717:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.560719:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.560721:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2735c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.560725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.560737:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.560747:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.569283:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.569395:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.569397:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.569399:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2735e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.569401:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.569414:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.569424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.579172:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.579291:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.579293:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.579295:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27360 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.579297:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.579314:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.579325:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.587726:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.587848:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.587850:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.587852:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27362 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.587855:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.587872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.587882:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.596326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.596453:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.596454:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.596457:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27364 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.596459:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.596478:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.596488:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.605191:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.605299:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.605300:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.605303:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27366 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.605305:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.605317:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.605326:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.614537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.614646:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.614648:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.614651:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27368 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.614653:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.614665:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.614675:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.623748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.623869:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.623871:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.623873:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2736a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.623875:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.623894:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.623904:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.633208:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.633331:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.633333:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.633335:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2736c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.633337:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.633367:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.633378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.641762:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.641885:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.641887:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.641889:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2736e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.641891:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.641910:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.641921:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.651091:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.651199:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.651201:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.651203:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27370 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.651205:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.651222:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.651232:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.660756:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.660868:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.660870:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.660872:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27372 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.660874:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.660886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.660901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.669481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.669626:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.669629:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.669632:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27374 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.669634:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.669653:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.669663:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.678787:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.678912:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.678913:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.678916:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27376 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.678918:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.678937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.678947:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.688374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.688494:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.688496:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.688498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27378 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.688501:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.688523:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.688533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.696768:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.696879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.696881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.696883:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2737a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.696885:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.696901:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.696911:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.705401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.705514:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.705515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.705518:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2737c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.705520:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.705536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.705545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.713820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.713942:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.713944:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.713946:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2737e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.713949:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.713964:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.713975:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.723548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.723665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.723667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.723669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27380 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.723671:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.723687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.723697:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.732153:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.732259:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.732261:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.732263:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27382 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.732265:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.732280:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.732290:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.740708:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.740828:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.740829:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.740832:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27384 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.740834:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.740850:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.740860:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.749379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.749490:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.749492:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.749494:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27386 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.749496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.749519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.749528:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.758849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.758960:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.758961:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.758964:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27388 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.758966:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.758980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.758988:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.768336:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.768451:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.768453:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.768455:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2738a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.768458:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.768469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.768479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.776633:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.776752:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.776754:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.776756:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2738c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.776759:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.776778:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.776793:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.785677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.785787:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.785789:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.785791:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2738e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.785793:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.785804:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.785814:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.798553:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.798672:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.798673:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.798676:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27390 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.798678:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.798695:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.798706:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.806891:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.807003:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.807005:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.807007:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27392 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.807009:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.807026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.807036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.815244:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.815359:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.815361:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.815363:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27394 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.815365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.815377:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.815387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.824488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.824622:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.824624:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.824627:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27396 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.824631:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.824653:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.824664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.834093:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.834217:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.834219:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.834221:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27398 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.834223:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.834239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.834249:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.845381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.845479:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.845481:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.845483:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2739a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.845485:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.845501:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.845518:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.856156:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.856269:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.856271:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.856273:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2739c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.856275:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.856291:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.856301:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.866978:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.867106:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.867108:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.867110:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2739e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.867113:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.867133:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.867143:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.877636:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.877750:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.877752:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.877755:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.877757:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.877767:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.877777:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.888176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.888282:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.888284:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.888288:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.888290:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.888305:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.888314:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.899322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.899600:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.899602:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.899604:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.899606:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.899623:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.899634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.910299:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.910424:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.910426:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.910428:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.910430:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.910442:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.910458:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.921248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.921363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.921365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.921367:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.921369:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.921381:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.921391:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.932478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.932710:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.932711:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.932713:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.932716:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.932731:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.932742:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.943449:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.943569:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.943571:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.943573:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.943576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.943593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.943603:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.952654:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.952771:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.952774:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.952776:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.952778:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.952794:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.952804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.963423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.963533:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.963535:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.963537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.963539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.963558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.963569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.974318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.974451:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.974453:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.974455:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.974458:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.974474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.974484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.985267:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.985372:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.985374:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.985376:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.985378:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.985389:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.985398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773279.994781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773279.995194:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773279.995196:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773279.995198:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773279.995201:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773279.995216:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773279/real 1521773279]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773291 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773279.995226:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.006241:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.006363:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.006365:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.006367:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.006369:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.006381:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.006391:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.017084:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.017197:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.017199:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.017201:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.017203:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.017219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.017229:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.027631:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.027754:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.027756:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.027758:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.027760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.027776:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.027786:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.038227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.038334:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.038336:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.038338:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.038340:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.038363:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.038372:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.047964:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.048103:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.048104:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.048106:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.048109:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.048125:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.048141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.058777:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.058895:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.058897:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.058899:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.058901:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.058918:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.058928:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.068229:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.068344:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.068357:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.068359:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.068361:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.068381:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.068392:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.078952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.079063:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.079065:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.079067:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.079069:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.079085:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.079095:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.089618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.089743:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.089745:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.089747:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.089749:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.089765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.089775:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.098515:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.098627:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.098629:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.098631:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.098633:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.098644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.098654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.109482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.109617:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.109619:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.109623:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.109626:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.109639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.109649:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.119227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.119358:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.119360:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.119362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.119365:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.119387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.119397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.128458:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.128582:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.128584:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.128586:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.128590:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.128604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.128613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.139214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.139338:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.139340:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.139343:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.139349:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.139374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.139384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.152856:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.152987:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.152989:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.152991:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.152994:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.153017:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.153027:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.163624:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.163753:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.163755:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.163757:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.163760:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.163787:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.163797:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.172831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.172954:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.172956:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.172958:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.172961:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.172982:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.172992:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.183806:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.183934:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.183936:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.183939:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.183941:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.183969:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.183984:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.194801:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.194928:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.194929:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.194933:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.194936:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.194963:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.194973:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.204808:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.204936:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.204938:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.204940:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.204942:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.204964:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.204974:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.215733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.215854:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.215856:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.215858:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.215861:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.215880:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.215890:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.225084:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.225199:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.225201:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.225203:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.225205:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.225221:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.225231:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.234546:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.234661:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.234663:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.234665:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.234667:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.234684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.234694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.245123:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.245246:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.245248:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.245250:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.245252:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.245271:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.245282:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.255814:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.255932:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.255935:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.255937:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.255939:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.255960:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.255971:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.265565:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.265687:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.265688:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.265691:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.265693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.265709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.265719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.276210:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.276330:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.276332:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.276334:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.276336:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.276367:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.276378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.286817:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.286928:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.286930:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.286932:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.286934:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.286949:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.286958:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.297541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.297662:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.297664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.297666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.297668:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.297689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.297699:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.308370:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.308489:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.308491:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.308493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.308495:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.308524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.308532:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.319209:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.319339:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.319341:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.319343:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.319356:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.319376:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.319390:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.326459:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.326580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.326582:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.326584:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.326586:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.326604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.326614:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.337244:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.337367:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.337369:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.337372:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.337374:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.337396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.337405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.347778:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.347896:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.347898:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.347900:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.347902:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.347922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.347932:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.358624:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.358759:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.358761:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.358763:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.358766:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.358786:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.358796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.368102:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.368217:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.368219:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.368221:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x273fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.368223:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.368239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.368250:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.378643:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.378761:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.378763:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.378765:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27400 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.378767:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.378787:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.378798:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.388259:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.388376:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.388377:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.388380:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27402 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.388382:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.388394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.388404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.397460:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.397578:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.397580:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.397582:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27404 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.397584:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.397602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.397613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.408046:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.408157:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.408159:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.408161:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27406 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.408163:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.408178:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.408188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.418730:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.418852:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.418854:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.418856:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27408 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.418858:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.418873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.418882:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.428330:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.428450:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.428452:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.428454:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2740a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.428457:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.428473:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.428483:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.438865:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.438978:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.438980:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.438982:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2740c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.438984:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.439004:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.439014:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.447871:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.447987:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.447989:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.447991:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2740e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.447993:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.448008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.448024:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.457273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.457375:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.457377:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.457379:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27410 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.457381:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.457393:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.457402:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.467026:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.467155:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.467157:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.467160:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27412 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.467162:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.467182:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.467192:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.477843:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.477961:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.477962:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.477965:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27414 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.477967:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.477987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.477997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.487763:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.487880:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.487881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.487885:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27416 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.487887:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.487903:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.487913:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.499031:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.499179:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.499180:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.499183:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27418 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.499185:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.499206:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.499216:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.510177:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.510294:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.510296:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.510298:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2741a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.510300:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.510320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.510330:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.521209:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.521337:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.521338:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.521340:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2741c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.521343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.521374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.521385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.531709:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.531833:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.531835:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.531837:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2741e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.531839:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.531859:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.531869:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.541289:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.541407:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.541408:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.541411:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27420 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.541413:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.541429:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.541439:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.552208:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.552652:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.552655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.552657:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27422 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.552659:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.552682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.552691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.562380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.562488:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.562489:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.562491:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27424 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.562494:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.562504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.562520:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.573229:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.573341:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.573342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.573355:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27426 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.573357:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.573378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.573388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.582491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.582613:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.582615:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.582617:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27428 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.582619:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.582629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.582645:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.591682:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.591796:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.591797:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.591799:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2742a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.591801:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.591813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.591822:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.601373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.601490:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.601491:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.601493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2742c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.601496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.601518:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.601527:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.612548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.612794:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.612796:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.612798:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2742e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.612800:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.612820:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.612831:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.623560:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.623676:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.623678:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.623680:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27430 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.623682:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.623702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.623712:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.634015:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.634163:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.634165:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.634167:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27432 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.634169:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.634184:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.634194:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.644582:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.644703:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.644705:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.644707:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27434 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.644709:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.644724:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.644734:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.655532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.655653:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.655655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.655657:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27436 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.655659:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.655680:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.655690:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.664908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.665021:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.665022:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.665025:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27438 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.665027:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.665044:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.665054:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.674495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.674614:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.674616:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.674618:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2743a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.674620:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.674639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.674648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.683696:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.683802:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.683804:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.683806:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2743c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.683808:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.683818:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.683825:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.692736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.692841:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.692843:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.692845:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2743e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.692847:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.692858:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.692867:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.702549:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.702664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.702665:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.702668:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27440 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.702670:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.702687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.702697:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.711904:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.712018:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.712019:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.712022:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27442 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.712024:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.712039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.712055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.721669:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.721772:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.721773:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.721775:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27444 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.721778:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.721789:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.721799:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.730945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.731046:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.731048:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.731050:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27446 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.731052:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.731063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.731072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.740515:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.740624:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.740625:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.740628:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27448 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.740630:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.740639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.740648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.751260:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.751374:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.751376:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.751378:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2744a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.751380:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.751396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.751407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.761785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.761899:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.761901:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.761903:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2744c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.761905:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.761925:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.761935:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.772337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.772467:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.772469:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.772471:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2744e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.772473:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.772493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.772503:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.782802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.782936:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.782938:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.782941:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27450 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.782943:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.782963:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.782973:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.793280:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.793413:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.793415:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.793417:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27452 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.793420:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.793441:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.793451:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.804048:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.804166:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.804167:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.804170:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27454 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.804172:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.804191:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.804201:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.814567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.814694:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.814696:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.814698:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27456 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.814700:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.814720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.814730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.825453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.825595:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.825597:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.825599:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27458 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.825602:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.825621:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.825631:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.836139:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.836258:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.836260:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.836262:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2745a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.836265:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.836284:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.836295:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.846869:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.846988:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.846991:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.846993:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2745c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.846995:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.847015:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.847031:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.857534:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.857658:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.857659:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.857662:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2745e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.857664:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.857684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.857695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.868161:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.868280:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.868282:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.868284:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27460 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.868286:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.868305:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.868316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.879009:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.879173:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.879175:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.879177:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27462 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.879180:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.879200:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.879211:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.889757:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.890272:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.890274:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.890276:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27464 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.890278:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.890298:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.890307:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.900863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.900992:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.900993:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.901031:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27466 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.901034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.901054:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.901065:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.911530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.911661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.911663:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.911665:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27468 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.911667:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.911688:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.911698:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.922175:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.922291:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.922293:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.922295:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2746a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.922297:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.922317:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.922327:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.932727:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.932847:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.932849:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.932851:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2746c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.932853:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.932873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.932883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.943181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.943305:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.943307:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.943309:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2746e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.943311:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.943331:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.943341:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.953873:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.953985:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.953987:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.953989:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27470 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.953991:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.954011:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.954021:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.964409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.964529:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.964530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.964533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27472 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.964535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.964561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.964571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.973840:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.973958:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.973960:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.973962:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27474 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.973964:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.973979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.973990:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.984610:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.984738:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.984740:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.984742:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27476 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.984745:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.984766:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.984782:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773280.995531:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773280.995661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773280.995662:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773280.995665:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27478 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773280.995667:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773280.995681:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773280/real 1521773280]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773292 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773280.995691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.006499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.006626:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.006628:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.006630:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2747a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.006632:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.006654:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.006664:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.016224:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.016338:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.016340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.016342:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2747c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.016344:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.016369:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.016380:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.026827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.026949:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.026950:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.026952:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2747e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.026956:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.026975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.026986:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.037715:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.037837:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.037838:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.037841:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27480 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.037843:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.037858:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.037868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.048207:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.048335:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.048337:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.048339:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27482 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.048341:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.048371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.048382:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.059010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.059124:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.059125:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.059128:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27484 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.059130:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.059149:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.059160:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.069495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.069630:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.069631:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.069634:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27486 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.069636:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.069651:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.069661:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.079106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.079220:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.079221:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.079224:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27488 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.079226:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.079241:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.079250:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.090375:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.090486:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.090488:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.090491:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2748a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.090493:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.090520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.090529:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.101180:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.101300:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.101301:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.101304:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2748c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.101306:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.101325:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.101336:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.111887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.112012:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.112013:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.112015:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2748e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.112018:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.112039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.112050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.123398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.123501:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.123503:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.123505:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27490 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.123513:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.123536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.123551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.133940:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.134057:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.134058:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.134060:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27492 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.134062:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.134083:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.134093:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.144812:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.144935:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.144937:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.144940:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27494 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.144943:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.144963:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.144973:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.155805:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.155930:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.155945:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.155947:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27496 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.155950:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.155977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.155987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.166758:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.166884:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.166886:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.166888:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27498 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.166891:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.166919:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.166929:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.177841:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.177967:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.177969:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.177971:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2749a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.177974:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.178001:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.178011:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.187437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.187556:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.187558:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.187561:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2749c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.187563:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.187577:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.187587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.198138:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.198255:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.198256:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.198259:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2749e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.198261:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.198287:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.198297:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.208848:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.208978:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.208979:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.208982:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.208984:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.209013:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.209023:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.218466:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.218586:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.218588:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.218590:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.218593:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.218607:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.218616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.227971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.228093:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.228095:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.228097:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.228100:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.228115:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.228125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.238655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.238777:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.238779:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.238781:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.238784:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.238800:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.238809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.248297:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.248416:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.248417:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.248420:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.248422:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.248439:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.248448:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.255755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.255926:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.255928:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.255930:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.255932:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.255949:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.255965:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.266642:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.266770:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.266771:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.266773:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.266776:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.266797:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.266807:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.277741:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.278264:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.278265:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.278268:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.278270:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.278289:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.278299:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.288222:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.288335:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.288336:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.288339:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.288341:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.288370:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.288380:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.298117:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.298223:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.298225:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.298227:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.298230:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.298245:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.298255:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.307412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.307540:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.307542:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.307545:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.307547:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.307569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.307580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.317320:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.317447:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.317448:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.317451:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.317453:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.317469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.317479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.328494:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.328640:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.328642:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.328645:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.328648:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.328668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.328678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.337830:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.337940:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.337942:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.337944:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.337946:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.337961:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.337971:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.348429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.348551:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.348553:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.348556:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.348558:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.348588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.348599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.359134:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.359257:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.359258:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.359261:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.359263:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.359282:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.359293:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.369955:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.370066:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.370068:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.370070:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.370072:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.370086:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.370096:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.380294:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.380425:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.380427:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.380429:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.380431:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.380448:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.380458:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.389748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.389860:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.389862:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.389865:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.389868:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.389883:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.389898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.398844:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.398961:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.398963:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.398965:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.398968:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.398983:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.398994:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.409466:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.409591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.409592:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.409595:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.409597:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.409617:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.409627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.420394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.420517:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.420519:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.420521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.420523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.420548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.420558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.431311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.431445:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.431447:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.431450:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.431452:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.431471:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.431482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.442251:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.442363:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.442364:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.442367:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.442369:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.442391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.442401:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.453127:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.453238:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.453241:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.453243:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.453245:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.453265:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.453276:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.462371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.462485:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.462486:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.462489:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.462491:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.462514:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.462525:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.473204:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.473331:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.473332:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.473335:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.473337:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.473369:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.473380:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.482396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.482516:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.482518:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.482520:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.482522:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.482542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.482552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.493289:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.493417:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.493419:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.493421:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.493423:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.493443:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.493454:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.503853:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.503970:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.503972:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.503974:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.503976:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.503995:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.504006:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.515251:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.515366:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.515368:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.515370:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.515372:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.515388:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.515398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.525562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.525677:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.525678:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.525681:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.525683:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.525698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.525713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.536548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.536679:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.536680:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.536682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.536684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.536705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.536716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.547375:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.547491:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.547492:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.547495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.547497:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.547525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.547534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.557389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.557512:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.557514:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.557516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.557518:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.557534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.557543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.568364:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.568484:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.568486:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.568488:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.568490:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.568520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.568531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.579242:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.579366:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.579368:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.579370:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.579372:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.579394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.579405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.592425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.592546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.592548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.592551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.592554:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.592584:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.592595:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.603174:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.603297:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.603298:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.603301:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.603303:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.603322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.603332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.613146:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.613264:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.613266:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.613268:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.613270:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.613284:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.613293:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.623805:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.623930:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.623932:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.623934:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.623936:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.623956:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.623966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.634535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.634658:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.634660:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.634662:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.634666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.634685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.634694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.643871:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.643988:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.643990:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.643992:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.643994:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.644009:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.644020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.654832:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.654952:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.654953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.654955:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.654958:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.654977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.654988:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.664816:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.664935:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.664937:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.664939:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.664941:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.664957:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.664972:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.674382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.674489:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.674491:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.674493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.674495:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.674513:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.674523:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.683771:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.683885:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.683887:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.683889:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.683891:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.683908:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.683918:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.692901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.693017:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.693019:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.693022:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x274fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.693024:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.693041:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.693052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.703433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.703559:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.703561:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.703563:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27500 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.703566:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.703593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.703604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.712745:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.713103:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.713105:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.713107:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27502 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.713109:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.713125:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.713136:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.722338:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.722472:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.722473:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.722476:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27504 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.722478:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.722493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.722504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.732082:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.732202:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.732204:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.732206:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27506 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.732208:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.732224:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.732234:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.741608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.741741:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.741743:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.741745:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27508 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.741748:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.741761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.741771:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.751746:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.751866:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.751870:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.751872:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2750a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.751874:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.751889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.751900:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.760949:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.761392:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.761394:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.761396:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2750c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.761398:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.761416:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.761427:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.772393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.772657:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.772659:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.772661:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2750e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.772663:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.772684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.772695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.783223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.783355:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.783357:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.783359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27510 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.783361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.783385:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.783395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.792598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.792725:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.792727:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.792729:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27512 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.792732:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.792747:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.792763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.803130:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.803252:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.803254:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.803256:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27514 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.803259:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.803278:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.803289:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.812541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.812697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.812698:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.812700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27516 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.812703:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.812714:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.812724:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.821769:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.821890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.821892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.821894:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27518 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.821897:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.821914:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.821924:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.832428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.832546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.832548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.832550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2751a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.832552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.832575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.832585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.843378:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.843492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.843493:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.843495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2751c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.843497:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.843519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.843528:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.853103:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.853214:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.853215:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.853218:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2751e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.853220:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.853235:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.853246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.863631:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.863759:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.863760:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.863763:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27520 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.863765:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.863786:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.863796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.873084:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.873196:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.873198:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.873200:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27522 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.873202:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.873218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.873228:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.883752:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.883873:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.883874:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.883876:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27524 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.883879:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.883899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.883910:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.892714:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.892832:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.892834:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.892836:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27526 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.892838:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.892853:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.892862:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.903216:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.903336:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.903337:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.903340:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27528 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.903342:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.903375:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.903386:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.912960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.913074:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.913076:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.913078:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2752a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.913080:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.913092:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.913102:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.922314:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.922444:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.922446:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.922448:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2752c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.922452:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.922467:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.922483:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.933298:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.933419:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.933421:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.933423:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2752e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.933425:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.933447:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.933457:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.943063:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.943205:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.943207:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.943210:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27530 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.943212:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.943235:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.943246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.952420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.952522:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.952524:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.952526:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27532 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.952528:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.952540:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.952550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.963158:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.963272:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.963274:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.963276:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27534 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.963279:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.963298:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.963308:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.972665:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.972782:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.972784:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.972786:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27536 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.972788:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.972800:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.972809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.982119:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.982255:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.982257:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.982260:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27538 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.982263:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.982278:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.982288:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773281.992839:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773281.992964:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773281.992966:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773281.992968:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2753a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773281.992970:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773281.992989:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773281/real 1521773281]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773293 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773281.993000:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.003585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.003714:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.003715:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.003718:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2753c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.003720:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.003741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.003752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.012798:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.012919:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.012921:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.012923:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2753e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.012925:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.012942:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.012952:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.023393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.023523:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.023525:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.023527:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27540 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.023530:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.023556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.023567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.032794:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.032910:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.032911:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.032914:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27542 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.032916:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.032933:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.032943:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.043563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.043690:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.043693:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.043695:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27544 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.043697:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.043717:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.043728:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.054410:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.054519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.054521:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.054523:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27546 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.054525:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.054548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.054564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.065334:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.065464:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.065466:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.065468:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27548 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.065471:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.065491:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.065502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.075325:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.075449:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.075451:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.075453:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2754a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.075455:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.075471:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.075481:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.084642:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.084754:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.084755:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.084757:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2754c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.084760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.084775:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.084784:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.094417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.094521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.094523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.094525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2754e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.094527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.094548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.094557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.105432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.105583:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.105585:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.105588:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27550 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.105590:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.105610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.105619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.114847:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.114962:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.114964:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.114966:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27552 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.114968:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.114978:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.114989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.124793:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.124909:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.124910:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.124912:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27554 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.124914:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.124931:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.124942:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.134083:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.134203:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.134205:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.134207:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27556 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.134209:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.134224:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.134235:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.144735:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.144856:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.144857:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.144860:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27558 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.144862:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.144882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.144892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.155672:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.155794:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.155796:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.155798:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2755a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.155800:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.155821:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.155832:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.166772:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.166887:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.166888:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.166891:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2755c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.166893:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.166914:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.166925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.174171:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.174284:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.174286:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.174288:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2755e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.174290:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.174306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.174316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.184874:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.184999:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.185001:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.185004:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27560 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.185006:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.185034:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.185049:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.194315:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.194450:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.194452:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.194454:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27562 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.194456:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.194478:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.194488:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.205298:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.205424:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.205426:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.205428:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27564 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.205431:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.205458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.205468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.216370:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.216478:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.216480:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.216482:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27566 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.216486:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.216516:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.216527:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.227398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.227520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.227522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.227525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27568 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.227527:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.227556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.227567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.238388:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.238532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.238534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.238537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2756a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.238540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.238567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.238578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.247748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.247841:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.247844:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2756c sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773282.247846:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.247861:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.247872:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.258308:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.258423:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.258425:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.258427:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2756e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.258429:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.258444:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.258455:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.268956:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.269085:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.269087:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.269089:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27570 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.269092:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.269111:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.269121:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.278775:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.278890:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.278891:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.278894:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27572 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.278898:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.278913:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.278924:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.289700:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.289797:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.289800:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27574 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773282.289802:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.289818:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.289828:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.299287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.299396:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.299398:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.299400:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27576 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.299402:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.299414:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.299424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.310248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.310369:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.310370:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.310373:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27578 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.310375:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.310397:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.310408:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.319518:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.319637:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.319638:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.319641:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2757a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.319643:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.319659:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.319675:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.330389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.330491:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.330492:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.330495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2757c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.330497:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.330518:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.330528:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.340736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.340835:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.340838:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2757e sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773282.340840:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.340858:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.340868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.351302:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.351422:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.351424:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.351426:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27580 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.351428:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.351451:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.351462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.360779:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.360894:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.360895:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.360897:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27582 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.360900:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.360916:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.360927:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.370288:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.370434:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.370436:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.370438:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27584 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.370440:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.370458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.370469:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.380748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.380862:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.380863:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.380866:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27586 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.380868:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.380887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.380897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.391621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.391740:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.391741:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.391744:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27588 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.391746:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.391766:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.391776:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.402231:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.402341:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.402342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.402355:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2758a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.402358:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.402380:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.402391:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.413182:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.413302:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.413304:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.413307:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2758c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.413309:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.413328:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.413339:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.422418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.422538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.422540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.422542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2758e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.422545:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.422568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.422579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.433239:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.433364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.433366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.433368:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27590 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.433371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.433392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.433403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.442760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.442879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.442881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.442883:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27592 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.442885:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.442895:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.442905:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.453480:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.453611:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.453613:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.453615:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27594 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.453617:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.453639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.453655:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.462555:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.462691:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.462693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.462697:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27596 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.462699:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.462722:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.462733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.472408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.472584:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.472586:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.472588:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27598 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.472590:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.472606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.472616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.482913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.483027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.483029:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.483031:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2759a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.483033:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.483054:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.483064:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.493562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.493682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.493684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.493686:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2759c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.493688:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.493708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.493719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.503079:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.503198:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.503199:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.503201:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2759e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.503204:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.503219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.503230:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.512552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.512663:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.512665:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.512667:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.512670:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.512685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.512695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.522230:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.522337:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.522340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.522343:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.522348:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.522372:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.522384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.531741:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.531851:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.531853:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.531855:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.531857:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.531872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.531883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.542727:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.542846:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.542848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.542850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.542852:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.542869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.542879:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.553740:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.553855:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.553857:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.553859:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.553862:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.553878:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.553889:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.563327:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.563453:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.563454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.563456:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.563459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.563475:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.563485:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.573657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.573770:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.573772:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.573774:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.573776:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.573793:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.573803:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.583366:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.583469:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.583471:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.583473:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.583475:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.583491:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.583517:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.592361:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.592466:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.592468:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.592470:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.592472:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.592483:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.592492:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.603141:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.603260:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.603262:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.603264:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.603266:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.603282:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.603292:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.612804:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.612920:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.612921:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.612924:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.612926:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.612941:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.612951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.623412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.623536:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.623538:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.623540:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.623543:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.623567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.623577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.633191:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.633307:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.633309:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.633311:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.633313:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.633330:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.633341:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.642742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.642865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.642867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.642869:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.642871:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.642887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.642897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.653621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.653735:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.653737:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.653739:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.653741:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.653757:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.653768:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.664785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.664912:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.664914:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.664916:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.664918:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.664937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.664947:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.676583:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.676705:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.676707:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.676709:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.676711:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.676730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.676741:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.688419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.688538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.688540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.688542:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.688544:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.688567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.688577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.699216:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.699340:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.699342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.699344:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.699358:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.699381:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.699392:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.710714:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.710836:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.710838:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.710840:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.710843:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.710858:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.710868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.721258:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.721371:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.721373:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.721375:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.721377:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.721388:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.721404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.732382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.732503:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.732505:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.732514:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.732516:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.732533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.732543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.741810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.741924:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.741926:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.741928:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.741930:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.741948:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.741958:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.752676:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.752794:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.752795:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.752797:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.752800:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.752815:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.752826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.763270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.763394:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.763396:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.763399:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.763401:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.763412:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.763422:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.772748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.772870:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.772872:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.772874:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.772876:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.772892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.772902:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.783273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.783410:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.783412:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.783415:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.783417:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.783440:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.783452:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.792692:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.793111:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.793113:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.793115:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.793117:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.793133:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.793144:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.803383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.803834:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.803836:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.803839:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.803841:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.803886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.803897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.815235:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.815366:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.815368:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.815370:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.815373:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.815392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.815403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.826219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.826341:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.826343:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.826357:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.826359:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.826379:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.826389:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.837075:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.837192:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.837194:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.837196:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.837198:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.837211:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.837220:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.847913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.848032:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.848034:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.848036:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.848038:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.848056:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.848067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.859143:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.859265:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.859267:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.859269:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.859271:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.859289:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.859304:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.870537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.870677:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.870679:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.870682:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.870684:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.870710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.870722:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.882320:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.882448:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.882450:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.882452:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.882454:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.882472:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.882482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.893537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.893659:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.893661:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.893663:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.893665:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.893683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.893693:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.905176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.905299:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.905300:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.905302:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.905305:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.905322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.905332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.915936:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.916048:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.916050:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.916052:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.916054:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.916066:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.916076:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.926389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.926498:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.926500:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.926502:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.926504:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.926525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.926534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.937619:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.937729:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.937730:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.937733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.937735:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.937746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.937756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.948708:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.948829:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.948831:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.948834:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.948836:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.948852:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.948862:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.960244:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.960366:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.960368:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.960370:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.960373:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.960393:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.960403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.972006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.972123:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.972125:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.972127:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.972130:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.972148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.972158:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.983267:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.983376:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.983378:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.983380:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.983382:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.983396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.983406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773282.994609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773282.994737:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773282.994738:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773282.994741:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773282.994743:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773282.994755:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773282/real 1521773282]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773294 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773282.994765:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.005388:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.005497:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.005499:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.005501:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.005503:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.005525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.005540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.016910:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.017023:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.017025:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.017027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x275fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.017031:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.017048:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.017059:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.028447:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.028578:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.028580:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.028583:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27600 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.028585:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.028601:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.028611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.040166:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.040285:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.040287:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.040289:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27602 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.040291:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.040309:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.040320:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.050704:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.050821:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.050823:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.050825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27604 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.050827:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.050846:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.050857:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.062522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.062645:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.062647:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.062649:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27606 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.062651:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.062670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.062680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.073101:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.073215:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.073217:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.073219:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27608 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.073221:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.073239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.073249:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.084673:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.084792:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.084794:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.084797:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2760a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.084799:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.084817:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.084827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.095524:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.095645:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.095646:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.095648:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2760c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.095651:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.095669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.095679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.107056:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.107168:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.107170:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.107172:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2760e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.107174:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.107195:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.107205:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.118221:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.118335:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.118337:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.118339:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27610 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.118341:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.118367:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.118377:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.129535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.129643:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.129645:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.129647:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27612 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.129649:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.129671:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.129680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.141217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.141340:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.141342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.141360:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27614 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.141363:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.141386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.141397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.151886:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.152016:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.152019:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.152022:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27616 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.152024:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.152046:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.152061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.159411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.159525:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.159527:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.159530:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27618 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.159532:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.159553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.159563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.170343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.170464:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.170466:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.170469:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2761a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.170471:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.170493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.170503:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.180264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.180389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.180391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.180394:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2761c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.180397:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.180420:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.180430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.191308:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.191445:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.191447:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.191449:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2761e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.191451:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.191472:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.191482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.202173:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.202298:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.202300:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.202302:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27620 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.202305:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.202326:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.202336:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.212829:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.212968:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.212970:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.212972:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27622 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.212975:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.212992:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.213002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.223554:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.223672:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.223674:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.223676:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27624 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.223678:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.223690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.223700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.235760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.235868:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.235869:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.235872:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27626 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.235874:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.235891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.235901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.247363:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.247479:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.247481:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.247483:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27628 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.247485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.247503:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.247523:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.258190:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.258304:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.258305:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.258308:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2762a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.258310:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.258327:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.258337:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.269602:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.269717:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.269718:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.269721:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2762c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.269723:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.269736:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.269746:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.281234:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.281360:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.281362:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.281364:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2762e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.281367:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.281384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.281394:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.292111:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.292226:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.292228:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.292230:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27630 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.292232:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.292249:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.292263:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.303395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.303494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.303496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.303498:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27632 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.303500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.303520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.303530:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.314733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.314857:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.314859:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.314861:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27634 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.314863:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.314881:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.314891:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.326425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.326532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.326534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.326536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27636 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.326538:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.326558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.326568:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.337860:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.337991:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.337993:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.337995:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27638 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.337998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.338016:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.338026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.349492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.349633:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.349635:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.349637:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2763a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.349639:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.349652:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.349661:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.360767:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.360883:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.360885:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.360887:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2763c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.360889:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.360908:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.360918:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.371755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.371863:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.371865:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.371867:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2763e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.371869:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.371886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.371897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.382627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.382755:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.382757:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.382759:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27640 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.382761:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.382778:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.382788:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.393774:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.393876:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.393878:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.393880:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27642 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.393882:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.393899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.393910:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.405181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.405290:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.405292:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.405296:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27644 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.405298:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.405315:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.405325:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.416603:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.416711:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.416713:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.416715:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27646 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.416718:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.416730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.416739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.427732:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.427859:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.427861:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.427863:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27648 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.427866:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.427881:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.427890:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.438484:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.438612:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.438614:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.438616:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2764a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.438619:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.438631:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.438647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.449240:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.449363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.449365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.449367:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2764c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.449369:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.449389:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.449400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.460717:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.460844:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.460846:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.460848:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2764e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.460850:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.460868:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.460878:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.471842:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.471956:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.471959:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.471961:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27650 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.471963:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.471982:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.471992:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.482577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.482694:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.482696:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.482698:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27652 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.482700:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.482719:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.482729:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.493453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.493576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.493577:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.493580:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27654 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.493582:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.493594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.493605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.504751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.504863:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.504864:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.504866:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27656 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.504869:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.504887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.504898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.516425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.516533:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.516535:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.516537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27658 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.516540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.516560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.516570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.527752:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.527869:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.527870:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.527873:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2765a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.527875:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.527892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.527903:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.538559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.538675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.538677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.538679:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2765c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.538681:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.538700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.538710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.549766:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.549880:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.549881:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.549884:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2765e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.549886:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.549905:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.549915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.561417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.561536:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.561538:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.561540:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27660 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.561543:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.561566:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.561576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.572217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.572356:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.572358:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.572361:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27662 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.572363:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.572378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.572388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.589342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.589468:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.589470:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.589473:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27664 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.589475:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.589499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.589522:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.600264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.600380:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.600382:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.600384:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27666 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.600386:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.600408:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.600419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.611151:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.611274:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.611275:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.611278:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27668 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.611280:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.611297:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.611308:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.621753:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.621863:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.621865:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.621867:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2766a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.621869:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.621886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.621897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.633224:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.633358:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.633360:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.633362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2766c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.633364:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.633379:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.633389:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.644473:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.644603:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.644604:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.644607:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2766e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.644609:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.644627:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.644638:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.656363:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.656475:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.656477:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.656479:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27670 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.656481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.656505:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.656524:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.667195:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.667313:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.667315:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.667317:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27672 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.667321:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.667338:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.667364:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.678287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.678415:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.678417:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.678419:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27674 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.678421:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.678436:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.678446:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.689219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.689328:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.689329:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.689332:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27676 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.689334:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.689364:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.689375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.699993:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.700100:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.700102:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.700104:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27678 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.700106:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.700124:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.700134:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.710755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.710874:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.710875:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.710878:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2767a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.710880:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.710896:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.710906:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.721597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.721716:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.721717:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.721719:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2767c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.721722:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.721732:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.721742:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.732417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.732532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.732534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.732537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2767e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.732540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.732559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.732575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.743153:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.743266:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.743268:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.743270:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27680 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.743273:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.743290:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.743300:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.754764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.754891:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.754893:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.754895:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27682 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.754897:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.754915:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.754925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.766368:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.766470:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.766472:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.766474:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27684 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.766476:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.766494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.766504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.776863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.776979:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.776981:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.776983:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27686 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.776985:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.777002:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.777013:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.788167:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.788283:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.788284:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.788286:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27688 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.788289:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.788306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.788316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.800016:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.800137:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.800140:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.800142:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2768a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.800144:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.800162:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.800173:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.810699:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.810824:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.810825:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.810828:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2768c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.810830:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.810847:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.810858:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.822440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.822553:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.822554:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.822557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2768e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.822559:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.822572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.822582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.833214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.833341:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.833343:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.833356:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27690 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.833358:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.833378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.833389:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.843733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.843852:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.843854:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.843856:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27692 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.843858:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.843870:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.843880:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.854987:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.855120:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.855121:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.855124:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27694 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.855126:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.855143:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.855153:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.866331:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.866466:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.866468:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.866470:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27696 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.866472:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.866490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.866500:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.877100:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.877210:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.877212:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.877214:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27698 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.877216:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.877228:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.877244:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.887490:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.887622:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.887624:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.887626:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2769a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.887628:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.887648:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.887658:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.898189:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.898315:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.898316:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.898319:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2769c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.898321:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.898338:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.898364:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.910101:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.910237:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.910238:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.910241:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2769e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.910243:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.910260:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.910271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.921924:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.922039:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.922041:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.922043:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.922045:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.922058:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.922068:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.933227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.933361:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.933362:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.933365:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.933367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.933387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.933397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.944384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.944515:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.944517:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.944519:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.944521:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.944549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.944560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.956303:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.956421:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.956423:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.956425:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.956427:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.956449:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.956459:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.968151:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.968263:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.968265:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.968267:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.968269:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.968296:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.968307:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.978741:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.978876:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.978878:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.978880:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.978882:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.978909:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.978918:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773283.990298:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773283.990418:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773283.990419:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773283.990421:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773283.990425:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773283.990447:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773283/real 1521773283]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773295 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773283.990458:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.001657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.001780:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.001782:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.001784:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.001786:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.001813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.001823:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.013129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.013243:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.013244:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.013247:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.013249:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.013265:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.013274:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.024994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.025117:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.025119:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.025121:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.025123:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.025151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.025167:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.036274:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.036389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.036391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.036393:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.036395:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.036409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.036417:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.046899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.047012:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.047014:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.047016:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.047018:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.047032:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.047040:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.058311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.058422:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.058424:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.058427:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.058429:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.058442:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.058452:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.069045:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.069185:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.069186:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.069189:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.069192:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.069220:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.069230:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.079810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.079931:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.079933:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.079935:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.079937:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.079964:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.079973:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.091383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.091484:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.091486:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.091488:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.091490:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.091515:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.091523:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.102112:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.102230:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.102232:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.102234:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.102236:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.102261:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.102270:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.112726:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.112858:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.112860:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.112862:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.112865:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.112892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.112903:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.123480:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.123616:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.123619:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.123621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.123624:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.123652:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.123662:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.134421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.134547:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.134549:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.134552:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.134554:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.134583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.134593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.151796:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.151914:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.151916:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.151918:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.151921:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.151942:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.151950:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.162417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.162528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.162530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.162532:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.162534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.162554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.162564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.169906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.170023:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.170025:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.170027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.170030:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.170045:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.170060:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.181747:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.181878:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.181879:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.181882:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.181884:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.181896:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.181906:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.194887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.195018:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.195020:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.195022:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.195025:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.195040:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.195050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.205549:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.205673:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.205675:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.205677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.205679:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.205698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.205709:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.216300:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.216412:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.216414:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.216416:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.216418:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.216437:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.216448:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.226896:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.227014:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.227015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.227018:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.227020:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.227037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.227047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.238319:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.238448:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.238450:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.238452:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.238454:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.238472:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.238482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.249274:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.249392:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.249394:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.249396:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.249398:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.249419:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.249430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.260505:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.260627:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.260629:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.260631:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.260634:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.260653:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.260663:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.271273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.271609:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.271611:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.271613:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.271616:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.271644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.271654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.282371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.282490:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.282491:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.282494:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.282496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.282520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.282530:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.293123:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.293249:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.293251:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.293253:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.293255:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.293273:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.293283:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.303850:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.304027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.304030:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.304033:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.304036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.304060:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.304070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.315382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.315495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.315497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.315499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.315503:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.315528:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.315543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.326307:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.326423:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.326425:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.326427:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.326429:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.326443:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.326453:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.336995:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.337109:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.337111:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.337113:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.337116:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.337134:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.337145:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.347455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.347591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.347593:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.347595:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.347598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.347614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.347624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.358637:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.358753:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.358755:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.358757:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.358759:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.358777:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.358788:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.369820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.369936:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.369937:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.369939:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.369942:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.369959:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.369969:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.381548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.381695:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.381697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.381700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.381702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.381721:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.381732:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.392429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.392554:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.392556:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.392559:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.392561:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.392589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.392600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.403720:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.403849:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.403850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.403853:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.403855:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.403873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.403883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.415480:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.415612:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.415613:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.415616:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.415618:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.415630:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.415640:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.426153:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.426272:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.426274:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.426276:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.426279:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.426295:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.426304:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.436896:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.437012:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.437013:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.437015:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.437018:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.437035:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.437045:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.447584:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.447697:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.447700:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.447702:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x276fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.447705:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.447725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.447736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.458185:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.458302:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.458303:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.458305:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27700 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.458307:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.458324:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.458340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.468754:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.468870:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.468872:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.468874:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27702 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.468876:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.468894:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.468904:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.479857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.479969:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.479971:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.479973:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27704 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.479975:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.479988:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.479997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.490703:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.490821:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.490822:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.490825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27706 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.490827:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.490844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.490855:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.501495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.501622:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.501624:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.501626:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27708 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.501628:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.501647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.501657:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.513294:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.513415:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.513417:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.513419:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2770a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.513421:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.513443:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.513454:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.523804:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.523921:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.523922:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.523924:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2770c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.523927:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.523945:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.523956:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.534895:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.535008:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.535009:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.535012:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2770e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.535014:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.535026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.535036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.545408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.545517:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.545518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.545520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27710 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.545522:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.545534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.545543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.556892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.557248:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.557250:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.557252:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27712 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.557254:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.557294:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.557305:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.567743:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.567846:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.567848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.567850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27714 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.567852:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.567870:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.567880:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.578337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.578457:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.578458:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.578461:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27716 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.578463:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.578480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.578491:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.591122:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.591523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.591524:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.591527:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27718 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.591529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.591549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.591559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.602453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.602573:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.602575:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.602577:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2771a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.602579:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.602591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.602607:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.613370:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.613474:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.613476:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.613478:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2771c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.613481:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.613498:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.613519:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.624126:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.624246:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.624248:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.624250:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2771e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.624253:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.624270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.624280:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.635020:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.635142:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.635144:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.635146:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27720 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.635149:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.635167:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.635177:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.646488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.646608:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.646609:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.646612:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27722 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.646614:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.646626:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.646636:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.658473:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.658609:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.658611:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.658613:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27724 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.658615:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.658634:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.658645:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.669488:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.669629:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.669631:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.669633:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27726 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.669635:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.669647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.669658:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.681007:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.681119:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.681121:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.681123:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27728 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.681125:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.681142:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.681153:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.692521:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.692649:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.692651:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.692653:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2772a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.692655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.692668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.692677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.703393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.703514:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.703515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.703519:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2772c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.703521:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.703539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.703549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.714396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.714520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.714522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.714525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2772e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.714527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.714550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.714560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.723696:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.723811:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.723813:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.723815:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27730 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.723818:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.723833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.723844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.732751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.732863:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.732865:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.732867:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27732 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.732869:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.732886:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.732896:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.743387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.743495:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.743497:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.743499:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27734 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.743501:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.743526:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.743541:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.753032:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.753169:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.753171:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.753174:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27736 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.753176:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.753198:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.753209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.762732:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.762864:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.762867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.762870:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27738 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.762872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.762887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.762898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.772542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.772650:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.772652:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.772654:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2773a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.772656:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.772671:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.772682:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.783383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.783500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.783501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.783504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2773c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.783513:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.783530:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.783540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.792844:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.792957:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.792958:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.792961:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2773e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.792963:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.792978:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.792988:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.802440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.802544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.802545:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.802548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27740 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.802550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.802561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.802571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.812268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.812384:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.812386:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.812388:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27742 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.812391:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.812407:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.812418:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.822888:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.823009:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.823011:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.823013:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27744 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.823015:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.823031:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.823041:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.832391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.832518:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.832520:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.832522:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27746 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.832525:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.832538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.832549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.843213:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.843355:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.843357:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.843360:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27748 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.843362:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.843374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.843385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.854190:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.854315:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.854316:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.854319:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2774a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.854321:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.854336:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.854362:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.865203:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.865327:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.865329:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.865331:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2774c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.865333:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.865362:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.865373:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.874702:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.874820:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.874822:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.874824:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2774e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.874826:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.874845:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.874860:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.885465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.885589:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.885591:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.885593:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27750 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.885595:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.885607:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.885617:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.896367:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.896475:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.896477:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.896479:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27752 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.896481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.896497:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.896514:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.905767:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.905882:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.905883:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.905885:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27754 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.905887:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.905904:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.905914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.916429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.916546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.916547:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.916549:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27756 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.916552:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.916571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.916581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.927818:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.927954:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.927955:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.927958:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27758 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.927960:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.927977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.927987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.939386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.939513:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.939515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.939517:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2775a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.939521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.939537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.939547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.950383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.950496:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.950498:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.950500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2775c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.950502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.950524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.950533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.961222:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.961338:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.961340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.961342:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2775e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.961344:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.961368:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.961379:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.972723:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.972848:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.972850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.972852:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27760 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.972854:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.972871:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.972881:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.984334:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.984459:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.984461:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.984463:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27762 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.984465:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.984481:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.984491:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773284.996005:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773284.996159:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773284.996161:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773284.996163:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27764 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773284.996165:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773284.996182:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773284/real 1521773284]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773296 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773284.996192:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.006772:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.006890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.006892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.006895:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27766 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.006897:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.006914:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.006925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.017594:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.017723:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.017725:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.017727:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27768 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.017729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.017739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.017755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.029382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.029512:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.029514:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.029516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2776a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.029518:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.029535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.029545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.041749:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.041879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.041881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.041883:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2776c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.041885:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.041902:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.041912:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.052698:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.052820:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.052822:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.052824:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2776e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.052826:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.052843:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.052853:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.064332:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.064444:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.064445:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.064448:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27770 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.064450:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.064465:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.064476:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.075751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.075900:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.075902:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.075905:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27772 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.075907:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.075924:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.075934:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.087595:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.087721:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.087723:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.087725:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27774 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.087727:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.087744:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.087755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.098230:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.098355:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.098356:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.098359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27776 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.098361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.098373:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.098384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.109765:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.109919:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.109921:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.109923:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27778 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.109925:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.109943:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.109953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.121230:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.121354:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.121356:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.121358:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2777a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.121360:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.121371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.121382:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.132712:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.132836:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.132837:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.132840:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2777c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.132842:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.132858:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.132868:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.144671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.144787:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.144789:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.144791:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2777e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.144793:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.144810:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.144820:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.153124:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.153239:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.153241:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.153243:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27780 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.153246:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.153269:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.153279:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.164078:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.164200:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.164202:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.164204:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27782 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.164206:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.164228:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.164243:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.175552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.175676:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.175678:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.175680:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27784 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.175683:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.175705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.175714:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.186425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.186559:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.186561:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.186564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27786 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.186566:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.186588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.186598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.197077:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.197200:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.197202:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.197204:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27788 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.197207:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.197229:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.197238:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.207450:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.207582:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.207584:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.207586:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2778a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.207589:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.207603:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.207612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.217773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.217909:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.217911:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.217913:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2778c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.217916:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.217938:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.217949:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.228484:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.228645:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.228647:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.228650:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2778e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.228652:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.228673:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.228683:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.239379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.239491:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.239493:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.239496:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27790 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.239498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.239524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.239534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.250264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.250378:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.250379:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.250382:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27792 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.250384:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.250396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.250406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.261044:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.261159:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.261161:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.261163:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27794 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.261166:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.261185:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.261195:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.271778:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.271903:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.271905:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.271907:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27796 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.271909:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.271928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.271939:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.282720:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.282836:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.282838:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.282840:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27798 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.282842:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.282854:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.282864:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.293748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.293859:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.293860:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.293862:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2779a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.293865:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.293881:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.293892:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.305340:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.305474:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.305475:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.305478:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2779c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.305480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.305497:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.305521:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.316766:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.316888:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.316889:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.316891:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2779e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.316893:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.316912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.316923:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.327644:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.327759:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.327761:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.327764:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.327767:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.327779:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.327789:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.338714:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.338832:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.338834:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.338836:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.338838:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.338856:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.338866:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.349947:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.350073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.350075:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.350077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.350079:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.350097:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.350108:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.361414:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.361523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.361525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.361527:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.361529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.361549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.361560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.372741:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.372857:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.372859:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.372861:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.372863:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.372881:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.372891:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.384875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.385000:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.385002:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.385004:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.385006:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.385024:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.385034:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.396076:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.396223:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.396226:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.396229:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.396231:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.396257:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.396269:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.407820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.407937:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.407939:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.407941:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.407943:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.407961:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.407971:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.418360:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.418474:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.418476:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.418478:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.418480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.418497:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.418514:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.429265:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.429394:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.429395:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.429398:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.429400:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.429420:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.429430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.440223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.440337:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.440339:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.440341:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.440343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.440371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.440381:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.451164:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.451499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.451501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.451503:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.451513:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.451534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.451550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.462745:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.462862:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.462863:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.462865:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.462868:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.462879:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.462890:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.473902:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.474018:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.474020:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.474022:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.474024:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.474042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.474052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.484771:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.484884:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.484886:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.484888:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.484891:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.484903:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.484913:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.496421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.496521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.496523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.496525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.496527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.496539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.496548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.507257:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.507368:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.507370:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.507372:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.507375:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.507391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.507400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.517983:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.518117:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.518119:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.518121:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.518124:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.518141:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.518150:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.528644:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.528777:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.528779:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.528781:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.528783:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.528801:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.528812:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.539699:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.539817:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.539818:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.539821:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.539823:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.539840:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.539850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.550211:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.550330:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.550332:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.550334:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.550337:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.550360:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.550370:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.560746:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.561065:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.561066:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.561069:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.561071:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.561110:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.561121:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.572365:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.572467:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.572469:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.572471:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.572473:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.572490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.572501:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.583111:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.583229:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.583230:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.583232:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.583236:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.583253:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.583263:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.594747:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.594877:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.594879:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.594882:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.594884:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.594899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.594914:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.605787:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.605910:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.605912:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.605914:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.605917:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.605936:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.605946:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.617546:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.617673:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.617675:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.617677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.617679:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.617691:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.617701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.628710:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.628835:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.628837:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.628839:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.628841:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.628860:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.628870:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.640493:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.640615:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.640616:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.640619:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.640621:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.640632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.640642:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.651380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.651500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.651501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.651505:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.651512:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.651530:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.651539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.662513:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.662634:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.662636:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.662638:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.662640:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.662652:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.662662:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.674097:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.674215:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.674217:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.674219:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.674221:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.674239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.674250:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.684787:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.684901:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.684903:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.684905:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.684907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.684926:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.684936:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.695849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.695968:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.695970:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.695972:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.695974:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.695992:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.696002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.706750:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.706876:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.706877:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.706880:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.706882:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.706900:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.706910:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.717750:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.717865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.717867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.717870:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.717872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.717888:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.717898:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.726774:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.726891:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.726893:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.726895:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.726897:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.726914:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.726924:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.736003:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.736115:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.736116:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.736119:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.736121:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.736131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.736146:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.746763:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.746886:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.746887:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.746889:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.746892:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.746908:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.746919:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.757457:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.757577:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.757578:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.757581:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.757582:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.757595:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.757605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.766441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.766543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.766544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.766547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.766548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.766560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.766569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.776380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.776505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.776512:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.776515:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.776517:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.776533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.776543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.787151:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.787267:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.787268:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.787270:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.787273:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.787288:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.787298:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.797775:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.797894:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.797896:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.797898:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.797901:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.797916:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.797927:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.807252:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.807361:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.807363:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.807365:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.807367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.807378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.807387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.817780:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.817894:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.817895:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.817898:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.817900:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.817917:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.817927:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.828653:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.828772:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.828774:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.828776:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.828778:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.828795:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.828805:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.839562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.839680:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.839682:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.839684:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x277fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.839686:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.839703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.839713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.850559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.850676:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.850678:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.850680:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27800 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.850682:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.850698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.850708:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.860122:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.860240:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.860242:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.860244:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27802 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.860247:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.860261:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.860271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.870739:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.870846:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.870848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.870850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27804 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.870852:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.870867:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.870880:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.881404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.881518:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.881520:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.881522:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27806 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.881524:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.881543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.881554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.892242:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.892365:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.892367:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.892369:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27808 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.892373:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.892390:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.892401:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.903163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.903298:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.903300:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.903302:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2780a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.903304:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.903320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.903330:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.914274:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.914397:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.914398:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.914401:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2780c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.914403:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.914421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.914431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.925118:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.925240:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.925242:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.925244:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2780e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.925246:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.925262:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.925272:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.934563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.934677:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.934678:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.934681:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27810 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.934683:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.934698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.934709:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.945393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.945501:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.945503:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.945505:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27812 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.945517:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.945536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.945546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.956303:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.956422:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.956424:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.956427:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27814 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.956429:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.956447:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.956458:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773285.967045:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773285.967170:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773285.967172:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773285.967174:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27816 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773285.967176:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773285.967191:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773285/real 1521773285]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773297 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773285.967202:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.223030:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.223194:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.223196:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.223198:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27818 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.223200:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773286.223218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773286.223228:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.232634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.232769:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.232771:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.232773:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2781a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.232776:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.232791:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.232804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.241747:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.241861:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.241863:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.241866:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2781c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.241869:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.241889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.241901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.252404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.252520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.252522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.252525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2781e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.252527:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.252543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.252553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.261868:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.261960:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.261962:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.261964:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27820 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.261966:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.261980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.261989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.272344:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.272458:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.272460:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.272462:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27822 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.272464:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.272476:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.272485:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.283168:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.283259:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.283261:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.283263:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27824 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.283265:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.283279:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.283288:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.292532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.292628:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.292629:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.292632:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27826 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.292634:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.292647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.292656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.302041:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.302148:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.302150:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.302153:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27828 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.302155:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.302171:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.302180:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.312751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.312852:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.312855:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.312857:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2782a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.312859:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.312874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.312882:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.322380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.322474:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.322476:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.322478:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2782c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.322480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.322490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.322499:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.333379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.333493:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.333495:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.333498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2782e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.333500:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.333524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.333534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.344652:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.344756:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.344757:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.344761:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27830 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.344763:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.344775:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.344784:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.355382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.355479:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.355480:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.355483:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27832 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.355485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.355497:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.355514:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.362672:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.362772:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.362774:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.362776:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27834 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.362778:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.362791:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.362800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.372249:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.372341:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.372343:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.372360:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27836 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.372363:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.372373:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.372381:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.382749:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.382849:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.382850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.382853:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27838 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.382855:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.382869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.382877:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.391947:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.392051:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.392052:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.392055:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2783a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.392057:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.392072:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.392080:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.402543:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.402648:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.402650:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.402652:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2783c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.402655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.402670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.402680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.412392:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.412491:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.412493:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.412495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2783e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.412497:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.412517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.412527:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.422834:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.422928:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.422930:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.422932:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27840 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.422935:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.422949:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.422958:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.433531:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.433638:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.433640:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.433642:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27842 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.433644:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.433659:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.433670:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.444737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.445025:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.445027:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.445029:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27844 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.445032:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.445047:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.445056:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.455532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.455638:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.455640:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.455643:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27846 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.455645:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.455660:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.455670:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.466373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.466474:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.466476:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.466478:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27848 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.466481:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.466493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.466518:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.477170:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.477289:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.477291:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.477294:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2784a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.477296:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.477309:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.477336:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.486752:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.486851:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.486852:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.486855:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2784c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.486857:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.486874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.486884:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.497931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.498031:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.498032:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.498035:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2784e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.498037:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.498051:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.498065:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.509009:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.509114:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.509116:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.509119:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27850 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.509121:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.509139:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.509149:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.519755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.519872:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.519874:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.519876:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27852 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.519878:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.519892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.519901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.530670:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.530794:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.530795:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.530798:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27854 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.530815:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.530828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.530861:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.540439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.540542:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.540545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.540548:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27856 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.540550:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.540567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.540577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.551560:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.551665:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.551667:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.551669:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27858 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.551672:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.551686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.551695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.561378:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.561476:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.561478:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.561480:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2785a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.561482:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.561494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.561514:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.571343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.571476:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.571478:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.571481:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2785c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.571499:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.571534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.571543:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.581228:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.581340:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.581342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.581362:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2785e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.581365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.581378:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.581404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.591260:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.591365:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.591366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.591384:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27860 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.591386:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.591399:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.591413:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.600854:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.600964:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.600966:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.600969:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27862 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.600971:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.600982:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.601009:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.612962:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.613089:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.613091:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.613093:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27864 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.613096:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.613107:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.613116:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.624376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.624492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.624493:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.624496:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27866 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.624498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.624534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.624549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.635770:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.635890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.635892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.635894:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27868 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.635896:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.635906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.635915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.647135:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.647260:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.647262:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.647264:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2786a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.647267:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.647279:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.647288:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.659396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.659502:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.659504:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.659512:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2786c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.659515:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.659546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.659562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.670325:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.670582:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.670584:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.670587:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2786e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.670589:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.670605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.670615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.681983:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.682106:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.682107:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.682110:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27870 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.682112:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.682125:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.682134:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.693436:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.693549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.693551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.693554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27872 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.693557:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.693572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.693588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.704928:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.705032:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.705034:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.705037:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27874 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.705054:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.705065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.705074:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.717055:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.717158:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.717160:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.717162:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27876 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.717164:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.717178:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.717187:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.727853:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.727954:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.727956:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.727958:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27878 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.727960:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.727975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.728005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.737454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.737569:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.737571:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.737574:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2787a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.737576:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.737595:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.737605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.747387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.747490:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.747492:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.747495:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2787c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.747498:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.747524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.747534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.758530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.758638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.758655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.758658:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2787e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.758660:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.758675:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.758691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.768761:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.768863:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.768865:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.768867:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27880 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.768870:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.768885:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.768894:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.778810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.778909:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.778910:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.778913:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27882 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.778915:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.778930:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.778955:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.790432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.790539:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.790541:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.790543:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27884 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.790546:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.790560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.790575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.802483:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.802653:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.802655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.802658:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27886 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.802660:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.802679:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.802691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.814540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.814655:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.814657:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.814660:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27888 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.814662:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.814677:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.814687:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.826495:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.826614:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.826616:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.826619:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2788a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.826621:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.826636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.826651:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.837852:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.837958:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.837960:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.837962:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2788c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.837964:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.837978:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.838003:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.849295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.849415:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.849417:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.849419:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2788e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.849422:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.849434:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.849444:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.860492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.860614:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.860616:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.860618:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27890 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.860620:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.860636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.860652:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.872118:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.872321:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.872323:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.872325:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27892 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.872328:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.872355:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.872384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.883524:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.883635:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.883637:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.883640:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27894 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.883642:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.883657:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.883666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.894616:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.894718:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.894720:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.894722:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27896 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.894725:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.894739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.894760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.906262:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.906375:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.906377:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.906379:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27898 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.906381:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.906394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.906403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.918010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.918118:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.918119:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.918121:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2789a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.918123:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.918136:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.918162:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.929569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.929673:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.929675:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.929677:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2789c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.929679:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.929694:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.929710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.940957:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.941082:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.941084:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.941086:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2789e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.941089:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.941104:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.941113:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.952445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.952554:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.952556:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.952558:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.952561:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.952576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.952585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.964404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.964521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.964523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.964526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.964528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.964543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.964558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.975577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.975688:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.975690:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.975692:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.975695:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.975709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.975718:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773286.987909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773286.988006:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773286.988008:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773286.988010:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773286.988012:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773286.988027:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773286.988036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.000163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.000338:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.000340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.000343:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.000348:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.000393:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773286/real 1521773286]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773298 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.000410:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.011620:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.011729:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.011731:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.011733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.011735:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.011751:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.011760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.022572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.022683:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.022685:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.022687:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.022690:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.022705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.022715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.034475:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.034585:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.034587:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.034589:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.034592:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.034607:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.034622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.046088:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.046190:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.046191:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.046194:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.046196:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.046226:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.046235:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.058031:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.058148:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.058150:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.058152:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.058154:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.058168:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.058177:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.069138:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.069255:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.069257:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.069259:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.069261:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.069276:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.069291:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.081593:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.081701:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.081702:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.081705:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.081707:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.081723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.081732:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.093344:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.093469:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.093471:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.093474:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.093476:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.093491:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.093500:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.105612:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.105719:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.105720:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.105723:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.105725:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.105740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.105756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.117987:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.118085:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.118087:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.118089:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.118091:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.118107:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.118116:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.129491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.129618:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.129620:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.129623:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.129626:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.129644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.129654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.141570:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.141681:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.141683:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.141686:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.141688:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.141704:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.141719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.152835:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.152955:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.152957:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.152974:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.152976:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.153005:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.153016:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.163959:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.164070:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.164072:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.164075:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.164078:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.164112:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.164122:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.175341:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.175461:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.175462:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.175465:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.175467:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.175479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.175493:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.186426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.186535:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.186537:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.186540:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.186542:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.186561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.186571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.196470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.196590:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.196592:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.196595:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.196598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.196619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.196630:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.207441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.207559:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.207561:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.207564:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.207566:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.207585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.207601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.216781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.216899:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.216901:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.216903:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.216905:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.216919:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.216928:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.227340:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.227472:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.227474:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.227476:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.227478:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.227490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.227499:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.237245:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.237357:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.237359:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.237362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.237364:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.237374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.237388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.246630:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.246725:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.246727:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.246729:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.246732:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.246746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.246755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.255887:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.255983:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.255985:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.255987:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.255990:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.256003:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.256012:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.266662:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.266761:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.266763:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.266766:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.266768:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.266797:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.266811:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.276538:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.276637:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.276639:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.276641:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.276643:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.276658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.276666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.287792:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.287912:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.287914:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.287916:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.287918:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.287932:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.287941:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.297389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.297485:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.297487:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.297489:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.297491:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:3.0:1521773287.297503:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:3.0:1521773287.297524:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.307382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.307494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.307496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.307498:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.307500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.307535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.307547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.317426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.317546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.317548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.317551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.317553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.317592:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.317604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.326952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.327066:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.327068:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.327070:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.327072:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.327102:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.327119:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.338060:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.338168:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.338170:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.338172:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.338175:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.338207:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.338217:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.345852:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.345946:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.345948:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.345950:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.345953:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.345986:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.345996:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.356909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.357052:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.357054:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.357057:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.357059:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.357090:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.357100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.367858:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.367964:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.367966:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.367968:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.367970:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.367987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.367997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.379008:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.379119:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.379121:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.379124:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.379126:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.379144:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.379154:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.390187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.390320:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.390321:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.390324:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.390326:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.390363:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.390375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.401542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.401656:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.401658:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.401660:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.401663:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.401696:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.401706:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.411380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.411491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.411493:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.411495:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.411497:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.411541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.411553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.422225:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.422330:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.422331:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.422334:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.422336:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.422373:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.422385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.433197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.433301:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.433303:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.433305:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.433308:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.433376:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.433387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.444654:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.444763:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.444764:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.444767:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.444769:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.444814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.444825:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.455547:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.455659:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.455661:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.455663:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.455666:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.455684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.455694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.466485:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.466611:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.466613:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.466615:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x278fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.466617:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.466649:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.466667:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.476483:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.476623:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.476625:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.476628:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27900 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.476632:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.476665:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.476675:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.486050:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.486160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.486162:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.486164:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27902 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.486167:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.486199:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.486209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.495568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.495695:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.495697:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.495699:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27904 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.495701:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.495733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.495744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.506613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.506726:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.506728:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.506730:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27906 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.506733:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.506763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.506790:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.517558:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.517671:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.517672:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.517675:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27908 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.517677:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.517709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.517719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.528550:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.528671:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.528673:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.528675:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2790a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.528677:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.528708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.528720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.538492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.538625:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.538628:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.538631:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2790c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.538634:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.538665:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.538677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.548395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.548504:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.548515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.548518:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2790e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.548520:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.548537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.548548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.557811:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.557938:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.557940:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.557942:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27910 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.557945:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.558008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.558020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.568766:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.568870:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.568872:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.568875:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27912 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.568877:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.568900:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.568926:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.579890:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.580012:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.580014:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.580016:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27914 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.580018:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.580047:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.580059:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.590398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.590505:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.590515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.590517:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27916 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.590520:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.590535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.590546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.601867:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.601980:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.601983:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.601985:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27918 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.601988:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.602020:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.602052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:3.0:1521773287.611389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.611497:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.611498:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.611501:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2791a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.611503:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.611585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.611597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.622403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.622529:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.622531:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.622533:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2791c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.622536:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.622566:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.622578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.632072:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.632198:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.632200:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.632203:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2791e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.632205:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.632221:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.632231:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.643455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.643586:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.643588:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.643590:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27920 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.643592:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.643619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.643630:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.653685:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.653812:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.653814:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.653817:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27922 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.653819:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.653834:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.653844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.664849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.664975:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.664977:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.664979:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27924 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.664982:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.664995:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.665005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.675259:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.675425:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.675427:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.675430:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27926 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.675432:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.675457:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.675467:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.686396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.686528:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.686530:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.686532:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27928 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.686535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.686568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.686579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.695875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.695977:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.695979:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.695982:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2792a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.695984:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.695998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.696008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.707075:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.707206:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.707208:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.707211:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2792c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.707213:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.707229:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.707239:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.718219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.718393:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.718395:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.718398:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2792e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.718400:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.718421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.718431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.727851:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.727986:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.727988:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.727991:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27930 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.727993:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.728015:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.728025:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.737386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.737491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.737494:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.737496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27932 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.737499:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.737523:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.737539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.747122:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.747244:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.747246:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.747249:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27934 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.747251:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.747270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.747280:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.758435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.758579:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.758581:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.758583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27936 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.758586:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.758629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.758640:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.768442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.768564:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.768565:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.768568:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27938 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.768570:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.768589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.768600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.779435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.779555:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.779557:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.779559:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2793a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.779562:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.779581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.779590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.788972:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.789075:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.789077:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.789080:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2793c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.789082:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.789096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.789106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.800270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.800401:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.800403:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.800405:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2793e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.800408:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.800422:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.800432:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.809930:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.810053:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.810055:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.810057:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27940 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.810060:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.810079:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.810089:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.820948:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.821087:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.821089:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.821091:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27942 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.821094:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.821115:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.821125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.830500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.830796:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.830798:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.830801:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27944 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.830803:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.830823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.830833:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.840730:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.840874:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.840876:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.840879:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27946 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.840881:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.840902:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.840912:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.850200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.850332:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.850334:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.850337:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27948 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.850339:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.850365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.850375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.861428:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.861560:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.861562:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.861564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2794a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.861567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.861598:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.861609:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.870844:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.870983:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.870985:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.870987:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2794c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.870989:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.871010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.871026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.882008:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.882137:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.882139:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.882142:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2794e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.882144:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.882164:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.882174:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.891487:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.891603:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.891605:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.891608:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27950 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.891610:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.891624:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.891634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.902410:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.902526:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.902528:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.902531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27952 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.902533:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.902555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.902565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.912147:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.912277:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.912279:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.912281:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27954 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.912284:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.912369:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.912381:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.922989:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.923096:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.923098:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.923101:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27956 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.923103:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.923118:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.923128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.933139:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.933279:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.933281:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.933283:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27958 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.933286:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.933306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.933317:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.944335:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.944500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.944502:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.944505:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2795a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.944515:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.944553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.944564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.954396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.954500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.954502:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.954505:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2795c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.954515:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.954547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.954557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.964522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.964638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.964640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.964643:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2795e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.964645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.964667:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.964677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.973827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.973945:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.973947:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.973950:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27960 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.973952:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.973973:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.973984:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.984537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.984665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.984667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.984670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27962 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.984672:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.984692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.984701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773287.995496:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773287.995651:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773287.995656:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773287.995661:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27964 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773287.995666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773287.995690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773287/real 1521773287]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773299 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773287.995700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.005421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.005540:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.005543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.005545:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27966 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.005548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.005572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.005588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.015180:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.015301:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.015303:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.015306:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27968 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.015309:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.015344:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.015363:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.025224:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.025338:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.025340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.025369:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2796a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.025371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.025386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.025396:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.036317:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.036418:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.036422:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2796c sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773288.036424:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.036441:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.036451:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.047369:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.047481:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.047483:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.047486:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2796e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.047488:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.047533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.047544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.056416:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.056522:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.056524:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.056526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27970 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.056529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.056543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.056554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.067517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.067654:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.067656:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.067659:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27972 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.067662:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.067684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.067694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.076768:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.076906:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.076910:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27974 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773288.076912:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.076926:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.076936:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.087601:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.087730:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.087732:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.087735:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27976 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.087737:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.087753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.087764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.097525:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.097633:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.097635:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.097639:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27978 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.097642:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.097655:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.097665:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.107334:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.107458:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.107460:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.107463:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2797a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.107465:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.107486:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.107496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.116909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.117079:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.117081:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.117084:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2797c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.117086:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.117113:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.117123:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.127712:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.127812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.127816:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2797e sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773288.127818:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.127840:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.127850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.137863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.137981:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.137983:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.137985:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27980 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.137988:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.138003:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.138018:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.148966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.149086:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.149088:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.149091:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27982 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.149093:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.149106:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.149116:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.160050:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.160176:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.160178:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.160181:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27984 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.160183:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.160216:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.160226:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.171184:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.171372:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.171374:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.171377:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27986 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.171379:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.171395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.171405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.180903:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.181079:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.181081:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.181084:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27988 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.181087:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.181107:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.181117:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.192054:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.192213:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.192215:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.192218:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2798a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.192220:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.192242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.192252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.203059:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.203171:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.203173:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.203176:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2798c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.203178:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.203207:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.203216:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.214157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.214310:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.214312:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.214315:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2798e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.214317:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.214337:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.214356:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.225587:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.225710:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.225712:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.225715:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27990 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.225717:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.225733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.225744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.235121:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.235241:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.235242:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.235245:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27992 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.235247:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.235327:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.235338:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.246274:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.246441:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.246443:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.246446:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27994 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.246448:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.246469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.246480:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.257465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.257587:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.257589:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.257592:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27996 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.257594:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.257608:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.257619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.268676:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.268812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.268814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.268816:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27998 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.268820:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.268842:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.268853:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.278906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.279077:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.279079:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.279082:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2799a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.279084:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.279105:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.279121:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.288585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.288710:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.288712:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.288714:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2799c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.288717:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.288730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.288741:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.297731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.297849:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.297850:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.297853:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2799e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.297855:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.297883:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.297891:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.308517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.308642:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.308644:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.308647:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.308649:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.308670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.308681:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.315737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.315904:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.315906:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.315909:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.315911:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.315933:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.315943:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.326814:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.326959:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.326961:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.326963:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.326966:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.326987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.326997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.336476:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.336594:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.336596:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.336599:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.336601:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.336614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.336625:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.345986:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.346106:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.346108:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.346110:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.346114:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.346135:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.346145:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.355977:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.356208:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.356212:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.356217:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.356222:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.356260:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.356281:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.365955:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.366086:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.366088:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.366091:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.366093:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.366114:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.366125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.376901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.377041:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.377043:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.377046:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.377048:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.377068:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.377077:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.386432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.386541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.386543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.386546:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.386548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.386572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.386582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.397484:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.397617:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.397620:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.397622:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.397625:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.397649:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.397659:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.408594:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.408757:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.408759:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.408762:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.408764:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.408803:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.408818:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.419661:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.419806:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.419808:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.419811:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.419813:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.419835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.419845:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.430404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.430526:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.430528:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.430531:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.430535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.430553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.430564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.439934:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.440047:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.440049:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.440051:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.440054:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.440069:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.440079:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.451130:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.451302:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.451305:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.451307:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.451310:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.451330:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.451340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.462246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.462384:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.462386:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.462389:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.462391:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.462410:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.462420:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.472733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.472842:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.472844:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.472847:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.472849:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.472874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.472901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.483419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.483541:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.483543:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.483546:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.483548:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.483616:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.483627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.494391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.494499:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.494501:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.494504:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.494513:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.494535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.494546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.504308:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.504417:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.504418:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.504421:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.504423:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.504438:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.504464:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.515504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.515649:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.515651:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.515654:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.515658:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.515678:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.515689:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.526745:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.526916:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.526918:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.526921:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.526923:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.526944:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.526955:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.537590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.537750:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.537752:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.537755:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.537757:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.537780:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.537790:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.547462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.547576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.547578:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.547581:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.547584:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.547608:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.547625:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.557409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.557561:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.557563:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.557566:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.557568:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.557633:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.557644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.567024:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.567130:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.567132:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.567135:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.567137:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.567166:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.567176:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.577402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.577525:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.577527:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.577530:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.577532:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.577550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.577561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.592446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.592554:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.592556:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.592558:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.592561:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.592580:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.592591:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.603482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.603653:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.603655:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.603658:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.603662:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.603730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.603742:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.614616:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.614753:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.614755:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.614758:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.614760:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.614798:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.614808:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.625413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.625519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.625521:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.625524:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.625527:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.625543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.625554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.635396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.635526:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.635527:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.635530:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.635532:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.635600:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.635612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.646533:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.646657:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.646659:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.646662:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.646665:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.646687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.646698:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.657946:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.658085:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.658087:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.658090:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.658092:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.658113:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.658123:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.669439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.669556:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.669558:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.669561:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.669563:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.669589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.669599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.679311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.679435:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.679437:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.679440:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.679442:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.679462:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.679472:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.689412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.689521:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.689523:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.689526:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.689530:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.689547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.689563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.700500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.700623:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.700625:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.700628:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.700630:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.700651:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.700661:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.710038:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.710160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.710162:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.710165:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.710167:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.710188:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.710198:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.721422:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.721564:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.721566:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.721569:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.721571:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.721638:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.721650:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.732465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.732597:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.732599:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.732602:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.732604:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.732623:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.732634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.743408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.743531:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.743534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.743536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.743539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.743561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.743571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.754394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.754500:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.754502:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.754505:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.754515:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.754531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.754542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.764383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.764499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.764501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.764504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.764512:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.764544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.764555:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.775402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.775520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.775522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.775525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.775528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.775551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.775562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.786394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.786528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.786530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.786533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.786535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.786571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.786583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.795530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.795682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.795684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.795687:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.795689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.795756:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.795768:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.806396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.806522:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.806524:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.806526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x279fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.806529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.806559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.806570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.817246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.817416:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.817418:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.817421:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.817423:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.817438:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.817449:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.826812:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.826947:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.826949:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.826952:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.826954:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.826976:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.826990:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.837517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.837855:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.837856:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.837859:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.837861:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.837926:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.837937:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.848822:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.848948:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.848950:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.848952:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.848955:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.848976:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.848986:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.859671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.860089:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.860107:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.860109:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.860112:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.860160:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.860171:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.871344:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.871483:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.871485:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.871488:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.871490:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.871533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.871544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.882456:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.882603:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.882605:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.882608:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.882610:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.882629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.882656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.891639:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.891847:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.891849:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.891851:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.891853:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.891874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.891885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.901866:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.902170:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.902172:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.902174:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.902177:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.902260:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.902271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.911342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.911475:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.911477:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.911480:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.911482:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.911503:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.911538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.921037:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.921199:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.921201:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.921204:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.921206:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.921227:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.921238:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.931797:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.931929:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.931931:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.931933:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.931936:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.931957:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.931967:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.941392:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.941634:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.941636:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.941638:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.941642:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.941666:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.941677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.952329:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.952478:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.952480:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.952483:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.952485:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.952505:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.952540:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.961802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.961939:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.961941:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.961944:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.961946:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.961966:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.961981:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.972597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.972722:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.972724:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.972727:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.972729:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.972750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.972760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.982360:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.982470:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.982472:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.982474:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.982477:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.982492:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.982502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773288.992437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773288.992597:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773288.992599:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773288.992602:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773288.992604:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773288.992622:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773288/real 1521773288]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773300 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773288.992633:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.002577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.002697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.002699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.002702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.002704:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.002718:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.002728:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.012223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.012344:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.012356:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.012359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.012361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.012374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.012384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.022009:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.022135:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.022137:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.022139:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.022143:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.022162:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.022172:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.032962:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.033091:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.033093:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.033096:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.033098:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.033119:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.033129:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.043013:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.043130:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.043132:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.043150:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.043153:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.043174:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.043184:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.052591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.052701:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.052702:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.052705:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.052707:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.052729:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.052739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.063691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.063805:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.063807:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.063810:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.063812:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.063833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.063843:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.074549:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.074688:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.074690:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.074693:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.074695:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.074716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.074726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.085516:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.085645:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.085647:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.085650:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.085653:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.085674:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.085685:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.094659:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.094769:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.094771:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.094774:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.094776:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.094790:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.094805:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.105375:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.105482:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.105484:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.105487:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.105491:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.105521:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.105532:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.116440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.116560:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.116562:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.116565:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.116567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.116586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.116596:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.127429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.127578:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.127580:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.127583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.127585:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.127627:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.127637:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.138433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.138569:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.138571:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.138573:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.138576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.138602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.138613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.149377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.149493:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.149495:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.149498:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.149500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.149531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.149541:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.160466:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.160597:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.160599:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.160602:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.160604:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.160620:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.160631:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.171492:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.171636:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.171638:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.171640:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.171643:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.171659:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.171670:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.182377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.182503:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.182514:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.182517:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.182519:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.182539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.182550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.193659:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.193840:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.193842:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.193845:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.193849:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.193887:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.193897:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.204976:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.205147:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.205150:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.205153:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.205156:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.205177:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.205187:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.214792:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.214900:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.214902:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.214905:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.214907:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.214928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.214938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.224914:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.225041:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.225044:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.225047:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.225049:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.225071:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.225081:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.236033:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.236484:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.236487:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.236490:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.236493:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.236525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.236542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.247403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.247515:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.247517:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.247519:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.247521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.247543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.247554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.256984:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.257103:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.257105:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.257108:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.257110:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.257131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.257141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.264588:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.264717:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.264719:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.264722:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.264725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.264746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.264757:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.274431:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.274551:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.274553:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.274556:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.274559:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.274583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.274594:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.284405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.284535:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.284537:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773289.284540:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a5a sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773289.284542:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.284560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.284570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.294320:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.294462:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.294464:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.294467:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.294469:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.294499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.294520:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.305768:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.305897:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.305899:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.305902:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.305905:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.305927:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.305937:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.316966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.317142:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.317144:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.317147:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.317150:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.317171:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.317182:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.327894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.328027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.328030:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773289.328033:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a62 sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773289.328036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.328056:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.328067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.337822:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.337950:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.337954:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773289.337957:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a64 sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773289.337960:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.337980:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.337991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.349280:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.349453:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.349455:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.349459:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.349461:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.349482:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.349492:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.360398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.360568:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.360570:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.360573:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.360575:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.360634:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.360646:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.371559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.371696:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.371698:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.371702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.371705:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.371727:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.371742:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.382697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.382814:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.382816:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.382819:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a6c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.382822:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.382835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.382845:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.393902:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.394062:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.394064:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.394067:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.394069:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.394089:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.394100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.404795:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.404923:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.404925:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.404928:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.404931:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.404952:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.404963:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.414752:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.414869:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.414871:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.414874:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.414876:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.414889:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.414900:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.426003:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.426113:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.426116:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.426119:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a74 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.426121:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.426142:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.426153:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.435396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.435544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.435546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773289.435549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a76 sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773289.435551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.435609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.435621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.446444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.446578:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.446580:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:2.0:1521773289.446583:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a78 sending 1 waiting 0: failed 5
00000800:00000100:2.0:1521773289.446585:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.446615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.446627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.455881:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.455995:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.455997:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.456000:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.456002:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.456021:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.456031:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.466476:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:2.0:1521773289.466624:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:2.0:1521773289.466626:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.466633:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.466636:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.466656:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.466667:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.477419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.477543:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.477545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.477548:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a7e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.477551:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.477573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.477583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.488393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.488525:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.488527:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.488530:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.488533:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.488548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.488559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.499477:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.499602:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.499604:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.499607:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.499610:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.499624:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.499634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.510679:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.510802:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.510804:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.510807:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.510809:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.510823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.510838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.521562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.521686:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.521688:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.521691:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.521693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.521709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.521719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.531371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.531500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.531503:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.531511:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.531514:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.531553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.531564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.542207:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.542355:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.542357:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.542359:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.542362:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.542374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.542385:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.553398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.553550:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.553552:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.553555:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.553557:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.553581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.553591:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.564383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.564538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.564540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.564543:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.564545:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.564576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.564588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.574457:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.574575:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.574577:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.574580:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.574582:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.574604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.574615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.585417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.585558:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.585560:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.585563:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.585565:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.585585:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.585596:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.594640:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.594755:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.594757:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.594760:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.594762:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.594800:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.594810:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.605559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.605682:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.605684:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.605687:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.605689:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.605705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.605715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.616615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.616797:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.616799:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.616802:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.616806:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.616828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.616838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.627781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.627902:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.627904:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.627907:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.627909:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.627922:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.627932:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.638931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.639108:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.639110:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.639113:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.639115:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.639137:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.639147:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.650387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.650487:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.650489:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.650492:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27a9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.650494:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.650520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.650536:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.659968:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.660096:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.660098:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.660100:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aa0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.660102:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.660123:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.660133:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.670120:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.670242:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.670244:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.670246:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aa2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.670249:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.670269:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.670279:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.681432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.681570:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.681572:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.681575:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aa4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.681578:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.681610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.681620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.692336:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.692469:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.692471:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.692474:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aa6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.692476:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.692496:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.692516:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.701820:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.701945:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.701947:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.701950:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aa8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.701953:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.701974:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.701984:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.712600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.712722:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.712724:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.712727:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aaa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.712729:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.712749:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.712760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.723759:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.723901:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.723903:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.723906:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.723908:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.723928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.723938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.735581:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.735694:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.735697:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.735699:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.735702:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.735718:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.735728:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.745334:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.745456:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.745458:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.745461:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ab0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.745463:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.745484:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.745494:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.755126:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.755270:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.755272:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.755275:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ab2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.755277:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.755300:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.755310:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.766401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.766544:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.766546:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.766549:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ab4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.766551:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.766575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.766585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.776125:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.776232:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.776234:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.776251:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ab6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.776254:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.776267:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.776277:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.786897:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.787011:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.787014:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.787016:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ab8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.787020:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.787033:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.787048:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.797583:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.797698:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.797700:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.797703:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.797705:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.797725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.797736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.808863:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.808983:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.808985:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.808988:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27abc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.808990:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.809012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.809022:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.819852:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.819984:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.819986:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.819988:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27abe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.819991:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.820012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.820022:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.831187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.831307:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.831309:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.831312:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ac0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.831314:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.831358:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.831369:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.842041:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.842231:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.842233:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.842235:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ac2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.842238:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.842259:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.842269:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.852208:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.852325:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.852327:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.852330:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ac4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.852333:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.852371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.852382:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.861467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.861578:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.861580:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.861583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ac6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.861585:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.861599:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.861610:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.871139:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.871265:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.871267:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.871270:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ac8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.871273:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.871293:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.871303:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.881818:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.881982:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.881984:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.881987:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.881989:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.882010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.882021:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.892499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.892628:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.892630:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.892633:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27acc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.892635:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.892649:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.892659:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.903318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.903460:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.903462:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.903465:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ace sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.903467:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.903486:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.903497:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.914422:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.914549:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.914551:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.914554:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ad0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.914556:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.914580:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.914590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.925725:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.925845:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.925847:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.925850:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ad2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.925852:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.925872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.925887:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.935361:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.935476:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.935478:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.935481:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ad4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.935484:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.935504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.935521:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.945223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.945363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.945365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.945367:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ad6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.945370:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.945385:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.945395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.956215:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.956342:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.956344:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.956358:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ad8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.956362:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.956382:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.956392:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.967142:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.967383:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.967385:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.967387:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ada sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.967390:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.967405:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.967416:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.978197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.978369:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.978371:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.978374:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27adc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.978376:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.978399:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.978410:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.988247:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.988363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.988365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.988368:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ade sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.988370:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.988385:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.988396:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773289.998044:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773289.998178:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773289.998180:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773289.998182:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ae0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773289.998185:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773289.998205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773289/real 1521773289]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773301 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773289.998215:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.008027:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.008159:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.008161:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.008164:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ae2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.008166:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.008187:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.008197:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.018545:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.018674:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.018676:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.018679:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ae4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.018681:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.018700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.018710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.028186:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.028298:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.028301:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.028303:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ae6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.028306:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.028321:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.028356:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.039420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.039558:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.039560:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.039563:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ae8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.039566:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.039607:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.039618:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.050448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.050592:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.050594:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.050596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.050599:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.050617:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.050628:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.060248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.060363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.060365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.060368:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.060370:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.060392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.060407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.071126:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.071258:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.071261:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.071263:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27aee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.071266:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.071285:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.071295:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.081713:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.081832:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.081834:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.081837:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27af0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.081839:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.081871:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.081881:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.093017:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.093172:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.093174:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.093177:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27af2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.093179:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.093199:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.093209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.104337:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.104466:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.104468:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.104470:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27af4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.104473:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.104493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.104503:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.115582:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.115728:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.115730:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.115733:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27af6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.115735:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.115774:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.115801:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.124907:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.125028:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.125030:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.125032:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27af8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.125036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.125057:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.125067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.135972:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.136085:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.136087:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.136090:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27afa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.136092:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.136113:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.136123:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.147168:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.147300:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.147302:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.147305:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27afc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.147307:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.147328:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.147338:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.156378:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.156494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.156496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.156499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27afe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.156501:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.156543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.156554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.167446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.167582:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.167584:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.167587:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.167590:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.167604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.167615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.178698:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.178825:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.178827:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.178830:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.178832:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.178854:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.178864:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.188369:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.188513:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.188530:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.188533:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.188535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.188556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.188567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.199642:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.199783:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.199785:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.199788:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.199790:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.199813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.199828:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.211629:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.211750:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.211753:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.211755:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.211759:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.211788:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.211798:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.219388:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.219498:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.219500:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.219503:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.219511:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.219550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.219561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.230681:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.230812:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.230814:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.230817:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.230819:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.230840:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.230850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.242017:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.242209:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.242211:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.242213:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.242216:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.242237:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.242247:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.251903:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.252006:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.252008:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.252011:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.252013:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.252026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.252036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.263039:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.263170:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.263172:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.263175:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.263177:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.263197:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.263207:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.274449:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.274587:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.274589:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.274592:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.274604:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.274640:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.274650:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.285936:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.286062:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.286064:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.286067:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.286069:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.286090:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.286100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.297655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.297774:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.297776:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.297779:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.297783:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.297796:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.297822:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.307481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.307591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.307593:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.307596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.307598:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.307612:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.307622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.316907:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.317023:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.317025:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.317028:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.317030:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.317050:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.317060:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.328522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.328736:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.328738:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.328741:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.328743:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.328765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.328776:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.338393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.338513:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.338515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.338518:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.338520:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.338548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.338565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.349760:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.349893:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.349895:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.349898:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.349900:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.349921:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.349931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.359754:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.359893:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.359895:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.359898:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.359900:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.359916:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.359925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.370627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.370760:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.370762:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.370765:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.370768:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.370805:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.370816:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.382452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.382575:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.382577:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.382580:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.382584:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.382609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.382619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.394192:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.394366:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.394368:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.394371:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.394373:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.394394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.394405:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.404808:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.404922:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.404924:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.404927:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.404929:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.404944:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.404954:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.415980:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.416110:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.416112:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.416115:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.416117:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.416138:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.416148:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.427617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.427742:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.427744:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.427747:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.427749:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.427771:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.427797:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.439277:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.439459:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.439461:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.439463:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.439466:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.439486:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.439496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.450654:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.450941:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.450943:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.450946:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.450948:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.450999:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.451009:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.461924:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.462028:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.462030:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.462033:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.462035:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.462048:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.462058:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.471943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.472149:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.472151:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.472153:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.472157:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.472179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.472189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.481776:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.481886:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.481888:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.481890:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.481893:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.481913:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.481928:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.491138:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.491255:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.491257:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.491260:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.491262:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.491282:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.491292:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.501010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.501133:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.501135:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.501138:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.501140:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.501169:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.501179:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.511817:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.511985:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.511987:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.511989:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.511992:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.512013:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.512023:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.521917:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.522037:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.522039:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.522042:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.522044:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.522066:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.522076:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.531943:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.532076:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.532078:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.532081:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.532083:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.532104:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.532114:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.541467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.541600:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.541602:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.541605:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.541607:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.541639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.541651:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.552408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.552566:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.552569:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.552572:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.552576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.552602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.552613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.563475:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.563624:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.563628:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.563634:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.563639:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.563683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.563694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.573627:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.573791:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.573793:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.573796:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.573814:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.573836:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.573846:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.584053:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.584172:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.584174:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.584176:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.584179:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.584198:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.584208:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.594187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.594357:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.594359:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.594362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.594364:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.594379:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.594390:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.604066:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.604196:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.604198:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.604200:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.604202:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.604223:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.604233:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.621897:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.622054:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.622056:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.622059:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.622061:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.622083:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.622099:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.632092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.632228:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.632229:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.632231:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.632234:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.632297:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.632308:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.642626:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.642783:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.642786:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.642788:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.642792:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.642829:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.642840:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.652213:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.652330:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.652332:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.652335:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b5a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.652337:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.652358:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.652369:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.661781:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.661890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.661892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.661895:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.661897:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.661910:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.661920:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.676586:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.676758:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.676760:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.676763:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.676766:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.676789:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.676800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.686657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.686782:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.686784:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.686787:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.686790:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.686803:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.686828:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.698195:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.698362:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.698364:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.698367:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b62 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.698369:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.698384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.698395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.708565:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.708682:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.708684:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.708687:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b64 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.708689:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.708710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.708721:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.719425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.719565:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.719567:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.719569:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.719572:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.719636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.719648:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.730926:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.731060:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.731062:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.731065:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.731068:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.731090:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.731100:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.740978:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.741384:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.741386:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.741388:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.741391:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.741413:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.741424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.752408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.752534:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.752536:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.752539:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b6c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.752541:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.752575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.752586:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.761867:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.762032:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.762034:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.762037:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.762039:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.762061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.762076:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.773737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.773871:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.773873:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.773876:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.773878:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.773899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.773909:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.784605:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.784731:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.784733:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.784736:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.784739:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.784760:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.784771:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.795249:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.795358:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.795359:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.795362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b74 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.795364:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.795387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.795397:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.805436:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.805591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.805593:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.805596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b76 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.805599:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.805616:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.805626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.814857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.814973:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.814974:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.814977:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b78 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.814981:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.814993:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.815003:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.826366:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.826494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.826496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.826499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.826501:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.826543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.826553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.836564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.836714:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.836716:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.836719:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.836721:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.836743:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.836754:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.847186:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.847295:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.847297:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.847300:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b7e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.847302:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.847322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.847332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.862478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.862595:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.862597:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.862600:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.862603:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.862616:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.862626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.872448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.872588:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.872590:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.872593:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.872595:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.872616:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.872627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.885020:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.885368:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.885370:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.885372:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.885375:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.885396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.885407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.896530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.896655:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.896657:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.896660:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.896662:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.896684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.896695:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.907948:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.908055:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.908057:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.908075:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.908078:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.908091:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.908106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.919381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.919500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.919502:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.919505:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.919516:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.919574:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.919587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.929800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.929910:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.929912:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.929915:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.929917:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.929938:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.929948:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.941374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.941499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.941501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.941504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.941514:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.941551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.941561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.951906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.952034:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.952036:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.952039:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.952042:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.952062:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.952070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.962918:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.963098:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.963100:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.963103:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.963105:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.963127:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.963137:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.974168:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.974304:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.974306:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.974308:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.974311:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.974330:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.974340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.985424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.985527:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.985529:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.985531:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.985534:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.985559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.985570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773290.995447:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773290.995612:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773290.995614:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773290.995617:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773290.995620:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773290.995656:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773290/real 1521773290]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773302 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773290.995666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.006532:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.006651:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.006653:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.006656:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.006659:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.006681:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.006691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.017637:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.017754:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.017756:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.017759:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.017762:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.017798:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.017808:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.028862:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.028996:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.028998:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.029001:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27b9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.029003:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.029023:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.029033:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.039992:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.040128:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.040130:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.040133:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ba0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.040135:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.040156:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.040166:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.049655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.049767:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.049769:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.049772:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ba2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.049774:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.049811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.049826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.060831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.060953:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.060955:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.060957:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ba4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.060960:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.060981:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.060991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.071595:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.071711:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.071713:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.071716:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ba6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.071719:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.071740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.071751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.082945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.083094:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.083096:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.083099:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ba8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.083103:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.083124:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.083134:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.093960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.094091:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.094093:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.094096:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27baa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.094098:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.094118:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.094128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.104899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.105022:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.105024:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.105027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.105029:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.105049:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.105059:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.114854:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.114963:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.114966:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.114975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.114977:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.114993:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.115003:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.124812:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.124931:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.124933:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.124935:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bb0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.124937:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.124956:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.124966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.134548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.134648:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.134650:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.134653:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bb2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.134655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.134669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.134679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.145600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.145723:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.145725:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.145728:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bb4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.145730:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.145752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.145763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.156859:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.156988:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.156990:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.156993:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bb6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.156995:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.157016:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.157026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.166853:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.166970:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.166972:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.166975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bb8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.166978:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.166998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.167009:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.177477:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.177599:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.177602:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.177604:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.177607:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.177626:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.177636:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.188658:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.188785:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.188787:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.188790:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bbc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.188793:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.188828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.188844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.196325:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.196452:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.196454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.196457:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bbe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.196459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.196479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.196490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.207445:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.207559:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.207562:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.207564:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bc0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.207567:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.207583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.207594:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.218894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.219061:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.219063:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.219066:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bc2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.219068:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.219088:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.219098:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.228893:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.229020:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.229022:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.229024:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bc4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.229027:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.229047:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.229057:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.239672:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.239845:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.239847:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.239850:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bc6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.239852:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.239873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.239883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.250742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.250913:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.250915:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.250917:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bc8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.250921:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.250942:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.250952:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.261967:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.262100:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.262102:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.262104:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.262107:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.262128:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.262138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.271960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.272086:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.272088:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.272091:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bcc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.272093:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.272114:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.272125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.282099:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.282223:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.282225:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.282228:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.282230:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.282264:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.282274:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.298361:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.298526:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.298528:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.298531:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bd0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.298533:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.298572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.298583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.309569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.309695:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.309697:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.309700:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bd2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.309702:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.309723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.309733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.321757:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.321879:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.321881:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.321883:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bd4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.321886:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.321923:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.321933:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.332671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.332810:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.332812:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.332815:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bd6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.332817:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.332856:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.332872:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.344372:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.344488:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.344490:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.344493:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bd8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.344496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.344522:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.344533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.356001:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.356183:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.356185:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.356188:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bda sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.356190:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.356230:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.356241:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.367755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.367894:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.367896:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.367899:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bdc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.367901:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.367940:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.367951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.379646:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.379767:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.379769:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.379772:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bde sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.379790:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.379824:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.379834:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.394729:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.394970:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.394972:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.394975:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27be0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.394977:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.395016:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.395027:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.406433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.406609:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.406611:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.406614:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27be2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.406617:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.406639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.406650:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.417796:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.417970:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.417972:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.417974:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27be4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.417977:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.418015:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.418026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.429228:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.429376:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.429378:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.429381:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27be6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.429383:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.429404:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.429415:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.440517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.440649:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.440651:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.440654:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27be8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.440658:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.440680:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.440691:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.452737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.452853:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.452855:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.452858:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.452861:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.452899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.452909:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.464537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.464661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.464663:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.464666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.464668:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.464685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.464696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.475921:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.476106:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.476108:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.476111:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.476113:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.476151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.476162:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.488318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.488445:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.488448:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.488450:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bf0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.488453:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.488474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.488490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.499539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.499679:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.499681:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.499684:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bf2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.499686:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.499709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.499720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.510866:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.511019:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.511021:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.511024:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bf4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.511027:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.511064:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.511075:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.522306:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.522471:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.522473:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.522476:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bf6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.522478:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.522525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.522535:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.534380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.534505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.534515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.534518:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bf8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.534537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.534560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.534570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.546841:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.546995:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.546997:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.547000:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bfa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.547002:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.547039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.547049:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.558829:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.558999:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.559001:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.559004:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bfc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.559006:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.559044:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.559055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.570895:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.571013:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.571015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.571018:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27bfe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.571020:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.571051:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.571061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.582294:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.582446:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.582448:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.582451:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.582453:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.582473:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.582484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.594386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.594492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.594494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.594497:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.594499:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.594531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.594542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.605680:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.605791:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.605793:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.605796:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.605814:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.605853:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.605863:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.616707:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.616812:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.616829:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.616832:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.616834:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.616866:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.616877:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.630590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.630766:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.630768:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.630771:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.630774:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.630796:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.630807:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.642726:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.642894:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.642896:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.642899:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.642901:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.642940:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.642956:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.654455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.654588:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.654590:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.654592:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.654595:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.654613:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.654624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.665777:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.665893:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.665895:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.665897:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.665900:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.665937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.665948:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.677899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.678063:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.678065:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.678068:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.678070:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.678107:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.678118:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.689542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.689653:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.689655:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.689658:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.689660:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.689683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.689693:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.701073:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.701221:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.701223:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.701226:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.701228:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.701266:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.701276:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.712914:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.713035:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.713037:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.713040:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.713042:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.713080:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.713091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.723998:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.724119:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.724121:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.724123:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.724127:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.724163:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.724174:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.736123:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.736255:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.736257:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.736260:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.736262:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.736297:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.736307:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.747441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.747566:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.747568:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.747571:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.747573:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.747593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.747604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.758653:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.758857:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.758859:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.758861:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.758864:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.758885:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.758895:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.771268:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.771415:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.771417:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.771420:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.771422:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.771445:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.771456:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.782264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.782397:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.782399:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.782402:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.782405:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.782421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.782431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.793409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.793568:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.793570:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.793573:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.793575:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.793610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.793627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.805077:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.805208:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.805210:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.805213:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.805215:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.805236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.805246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.816873:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.817057:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.817059:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.817062:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.817066:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.817087:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.817097:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.826519:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.826622:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.826624:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.826627:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.826629:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.826644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.826654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.836572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.836720:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.836722:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.836725:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.836728:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.836750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.836761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.846814:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.846946:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.846948:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.846951:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.846953:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.846973:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.846983:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.856368:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.856472:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.856474:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.856477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.856479:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.856499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.856534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.867404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.867579:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.867581:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.867584:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.867586:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.867610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.867620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.878750:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.878906:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.878908:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.878911:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.878913:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.878928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.878938:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.890421:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.890552:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.890554:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.890557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.890559:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.890586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.890596:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.901551:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.901658:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.901660:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.901663:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.901666:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.901682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.901692:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.912684:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.912824:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.912826:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.912829:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.912831:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.912852:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.912862:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.923827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.923970:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.923972:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.923974:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.923976:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.924010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.924020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.936288:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.936411:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.936413:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.936415:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.936418:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.936440:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.936456:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.947448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.947586:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.947588:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.947591:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.947593:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.947614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.947624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.958635:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.958813:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.958815:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.958818:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.958820:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.958841:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.958852:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.969498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.969623:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.969625:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.969628:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.969631:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.969645:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.969655:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.979606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.979733:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.979735:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.979738:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.979741:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.979763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.979789:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773291.991056:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773291.991177:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773291.991179:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773291.991182:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773291.991185:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773291.991207:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773291/real 1521773291]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773303 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773291.991217:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.002182:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.002363:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.002365:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.002367:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.002370:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.002389:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.002400:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.013819:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.013959:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.013961:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.013963:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.013966:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.013987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.013997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.025417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.025526:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.025528:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.025531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.025533:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.025558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.025569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.035617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.035785:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.035787:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.035790:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.035792:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.035814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.035824:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.046917:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.047051:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.047053:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.047056:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.047058:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.047079:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.047089:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.058449:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.058570:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.058572:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.058575:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.058577:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.058594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.058605:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.069517:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.069639:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.069641:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.069644:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.069646:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.069668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.069678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.079423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.079520:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.079522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.079525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.079529:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.079555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.079572:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.089429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.089560:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.089564:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.089570:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c5a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.089575:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.089607:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.089618:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.100660:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.100838:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.100840:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.100843:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.100845:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.100867:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.100877:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.112080:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.112183:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.112185:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.112188:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.112191:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.112204:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.112230:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.123539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.123702:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.123704:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.123707:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.123710:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.123736:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.123746:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.134642:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.134773:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.134775:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.134794:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c62 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.134796:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.134817:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.134827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.145973:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.146102:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.146104:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.146107:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c64 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.146109:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.146131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.146141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.157138:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.157322:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.157324:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.157327:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.157330:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.157376:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.157387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.168668:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.168803:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.168805:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.168808:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.168812:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.168825:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.168835:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.180062:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.180190:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.180192:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.180195:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.180197:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.180218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.180228:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.191462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.191595:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.191597:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.191600:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c6c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.191602:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.191622:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.191633:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.202476:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.202625:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.202627:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.202630:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.202632:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.202656:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.202667:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.213677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.213842:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.213844:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.213846:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.213849:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.213914:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.213925:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.221156:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.221259:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.221261:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.221264:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.221267:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.221280:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.221296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.232285:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.232419:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.232421:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.232424:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c74 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.232426:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.232452:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.232462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.241833:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.241963:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.241967:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.241973:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c76 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.241978:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.242001:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.242010:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.254397:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.254653:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.254655:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.254658:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c78 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.254662:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.254686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.254696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.264433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.264561:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.264563:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.264565:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.264568:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.264587:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.264597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.277669:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.277821:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.277823:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.277825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.277828:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.277850:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.277860:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.288519:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.288631:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.288633:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.288636:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c7e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.288638:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.288652:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.288663:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.299687:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.299818:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.299836:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.299839:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.299841:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.299857:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.299867:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.310908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.311033:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.311035:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.311038:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.311040:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.311061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.311072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.321875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.322048:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.322050:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.322053:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.322055:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.322077:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.322087:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.333455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.333583:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.333585:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.333588:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.333590:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.333609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.333620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.344738:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.344876:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.344878:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.344880:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.344884:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.344905:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.344915:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.356205:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.356380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.356382:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.356385:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.356387:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.356408:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.356419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.367424:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.367551:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.367553:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.367555:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.367558:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.367591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.367608:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.377217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.377539:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.377541:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.377544:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.377547:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.377571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.377581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.386735:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.386845:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.386847:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.386850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.386853:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.386882:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.386891:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.395858:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.396027:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.396029:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.396032:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.396034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.396055:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.396065:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.407558:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.407680:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.407682:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.407685:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.407688:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.407708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.407719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.418782:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.418913:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.418915:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.418917:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.418920:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.418941:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.418951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.429876:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.429994:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.429996:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.429999:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.430002:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.430023:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.430033:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.439883:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.440014:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.440016:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.440019:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.440021:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.440037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.440047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.450858:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.451002:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.451006:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.451012:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.451017:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.451039:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.451049:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.461598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.461777:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.461779:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.461782:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27c9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.461784:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.461822:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.461832:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.471415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.471523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.471525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.471528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ca0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.471531:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.471554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.471565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.482420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.482601:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.482603:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.482606:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ca2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.482608:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.482638:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.482650:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.492400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.492528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.492530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.492532:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ca4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.492534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.492564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.492575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.502378:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.502489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.502491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.502494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ca6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.502496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.502535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.502551:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.511888:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.512022:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.512024:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.512027:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ca8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.512030:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.512052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.512062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.523264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.523382:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.523384:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.523387:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27caa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.523389:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.523422:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.523432:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.533323:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.533722:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.533724:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.533727:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.533729:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.533752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.533763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.543665:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.543795:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.543797:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.543800:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.543802:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.543823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.543833:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.555219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.555343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.555377:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.555380:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cb0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.555382:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.555417:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.555428:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.566295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.566420:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.566422:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.566424:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cb2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.566427:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.566440:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.566450:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.577398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.577526:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.577528:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.577531:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cb4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.577533:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.577568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.577579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.587221:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.587358:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.587360:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.587362:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cb6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.587365:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.587379:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.587389:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.596639:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.596803:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.596805:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.596807:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cb8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.596811:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.596833:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.596843:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.606312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.606428:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.606430:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.606433:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.606435:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.606464:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.606474:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.615872:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.616028:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.616030:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.616033:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cbc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.616036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.616058:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.616068:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.626183:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.626299:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.626301:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.626304:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cbe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.626306:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.626325:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.626335:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.635731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.635872:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.635874:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.635877:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cc0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.635879:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.635901:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.635916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.646892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.647021:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.647023:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.647026:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cc2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.647028:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.647049:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.647059:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.656731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.656835:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.656837:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.656840:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cc4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.656842:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.656863:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.656874:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.666499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.666712:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.666714:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.666717:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cc6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.666719:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.666741:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.666751:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.676575:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.676729:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.676731:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.676734:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cc8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.676737:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.676759:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.676770:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.686425:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.686539:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.686541:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.686544:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.686546:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.686570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.686581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.697613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.697772:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.697774:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.697792:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ccc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.697794:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.697816:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.697827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.708937:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.709074:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.709076:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.709079:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.709081:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.709102:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.709112:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.718448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.718576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.718578:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.718580:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cd0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.718583:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.718645:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.718657:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.729930:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.730049:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.730051:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.730053:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cd2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.730056:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.730075:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.730085:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.739787:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.739910:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.739912:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.739915:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cd4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.739917:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.739936:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.739946:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.751077:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.751184:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.751186:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.751189:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cd6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.751191:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.751205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.751215:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.761972:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.762146:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.762148:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.762151:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cd8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.762155:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.762176:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.762186:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.773871:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.773982:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.773984:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.773987:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cda sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.773989:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.774010:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.774026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.784931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.785064:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.785066:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.785069:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cdc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.785071:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.785092:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.785102:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.794248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.794361:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.794364:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.794377:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cde sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.794379:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.794397:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.794408:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.804194:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.804382:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.804384:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.804387:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ce0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.804390:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.804411:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.804422:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.815591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.815716:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.815718:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.815721:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ce2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.815723:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.815745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.815756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.825397:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.825516:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.825518:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.825521:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ce4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.825523:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.825554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.825566:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.834818:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.835070:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.835072:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.835075:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ce6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.835077:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.835096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.835106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.844435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.844537:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.844539:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.844542:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ce8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.844546:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.844565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.844575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.854594:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.854715:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.854717:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.854720:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.854723:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.854745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.854755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.865808:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.865946:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.865948:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.865951:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.865953:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.865975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.865985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.877328:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.877486:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.877488:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.877491:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.877493:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.877522:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.877533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.887370:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.887675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.887677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.887680:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cf0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.887682:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.887702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.887713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.897443:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.897564:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.897566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.897568:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cf2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.897571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.897590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.897601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.908679:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.908812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.908814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.908816:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cf4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.908819:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.908835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.908851:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.919725:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.919864:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.919866:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.919869:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cf6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.919871:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.919891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.919901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.929580:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.929717:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.929720:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.929722:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cf8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.929726:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.929750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.929761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.940703:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.940840:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.940842:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.940844:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cfa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.940847:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.940869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.940879:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.951857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.951984:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.951986:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.951988:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cfc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.951991:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.952011:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.952021:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.962871:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.963043:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.963045:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.963048:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27cfe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.963050:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.963072:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.963082:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.974344:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.974462:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.974464:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.974467:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.974469:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.974490:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.974501:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.985657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.985816:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.985818:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.985821:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.985823:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.985844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.985854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773292.996953:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773292.997132:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773292.997134:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773292.997136:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773292.997139:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773292.997160:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773292/real 1521773292]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773304 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773292.997171:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.008455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.008591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.008593:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.008596:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.008598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.008619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.008629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.021535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.021647:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.021649:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.021652:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.021656:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.021669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.021678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.031168:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.031290:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.031292:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.031295:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.031297:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.031333:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.031343:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.042163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.042336:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.042338:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.042341:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.042343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.042387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.042398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.051800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.051900:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.051902:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.051905:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.051907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.051936:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.051951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.061375:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.061531:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.061533:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.061536:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.061538:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.061560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.061571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.070811:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.070930:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.070932:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.070935:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.070937:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.070958:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.070968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.081984:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.082176:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.082178:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.082181:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.082183:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.082205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.082215:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.091818:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.091951:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.091953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.091956:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.091958:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.091979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.091989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.101197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.101355:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.101356:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.101359:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.101362:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.101384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.101395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.111287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.111409:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.111411:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.111414:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.111416:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.111436:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.111447:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.122482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.122611:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.122613:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.122616:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.122618:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.122634:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.122645:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.133669:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.133843:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.133845:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.133848:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.133850:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.133871:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.133881:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.143609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.143728:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.143730:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.143733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.143735:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.143757:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.143768:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.153413:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.153531:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.153533:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.153536:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.153539:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.153553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.153563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.162957:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.163151:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.163153:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.163156:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.163158:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.163179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.163189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.170861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.170987:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.170989:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.170991:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.170994:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.171012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.171037:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.180204:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.180306:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.180309:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.180311:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.180315:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.180329:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.180344:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.189984:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.190133:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.190135:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.190137:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.190140:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.190161:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.190171:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.199624:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.199796:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.199798:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.199801:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.199803:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.199825:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.199835:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.210892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.211029:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.211031:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.211034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.211036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.211057:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.211067:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.221709:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.221831:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.221833:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.221835:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.221838:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.221859:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.221869:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.231473:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.231567:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.231569:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.231572:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.231574:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.231588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.231598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.242628:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.242784:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.242786:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.242788:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.242791:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.242828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.242839:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.253576:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.253702:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.253704:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.253707:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.253709:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.253725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.253735:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.263211:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.263334:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.263336:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.263339:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.263342:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.263367:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.263378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.272971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.273114:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.273116:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.273118:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.273121:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.273142:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.273152:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.283181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.283333:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.283335:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.283338:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.283340:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.283367:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.283378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.293107:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.293216:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.293218:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.293220:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.293223:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.293243:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.293253:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.304396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.304515:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.304517:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.304520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.304523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.304539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.304549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.315617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.315731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.315733:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.315736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.315738:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.315755:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.315771:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.324699:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.324813:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.324815:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.324818:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.324820:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.324836:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.324846:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.334400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.334505:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.334530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.334533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.334535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.334555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.334565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.344152:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.344268:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.344270:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.344273:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.344276:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.344313:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.344323:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.355478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.355600:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.355602:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.355605:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.355608:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.355630:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.355641:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.366668:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.366842:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.366844:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.366846:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.366849:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.366869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.366879:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.377109:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.377242:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.377244:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.377247:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.377249:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.377269:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.377279:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.388317:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.388444:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.388446:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.388449:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.388451:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.388471:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.388482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.398437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.398543:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.398545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.398548:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.398550:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.398568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.398579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.408069:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.408257:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.408259:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.408262:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.408264:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.408286:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.408296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.417981:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.418109:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.418111:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.418114:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.418116:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.418137:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.418147:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.429251:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.429364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.429366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.429369:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.429372:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.429409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.429420:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.441227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.441383:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.441385:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.441388:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d5a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.441390:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.441413:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.441424:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.450922:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.451043:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.451045:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.451048:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.451050:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.451071:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.451092:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.461575:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.461697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.461699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.461702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.461704:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.461726:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.461736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.471392:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.471494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.471496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.471499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.471501:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.471524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.471534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.480841:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.481016:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.481018:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.481021:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d62 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.481023:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.481045:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.481055:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.490456:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.490584:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.490586:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.490588:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d64 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.490591:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.490610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.490621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.501150:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.501267:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.501269:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.501272:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.501289:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.501309:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.501318:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.510481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.510591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.510594:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.510596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.510600:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.510614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.510625:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.521533:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.521669:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.521671:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.521674:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.521677:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.521691:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.521701:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.531418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.531494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.531497:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d6c sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773293.531500:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.531524:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.531534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.540903:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.541051:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.541053:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.541056:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.541058:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.541080:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.541090:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.550404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.550499:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.550501:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.550504:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.550515:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.550539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.550550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.561269:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.561389:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.561391:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.561394:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.561397:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.561411:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.561421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.572121:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.572215:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.572219:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d74 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773293.572221:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.572242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.572252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.584111:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.584205:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.584207:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.584209:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d76 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.584212:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.584225:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.584242:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.595444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.595554:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.595557:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.595559:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d78 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.595562:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.595586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.595597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.606482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.606600:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.606604:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.606606:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.606609:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.606630:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.606641:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.616381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.616532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.616534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.616537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.616540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.616562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.616573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.627589:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.627683:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.627686:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d7e sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773293.627689:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.627710:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.627719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.638749:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.638885:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.638887:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.638890:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.638892:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.638912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.638923:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.650111:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.650228:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.650230:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.650233:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.650235:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.650255:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.650265:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.661407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.661514:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.661516:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.661519:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.661521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.661539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.661549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.671365:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.671472:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.671474:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.671477:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.671479:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.671499:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.671533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.682018:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.682177:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.682179:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.682182:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.682184:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.682205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.682214:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.691258:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.691409:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.691411:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.691415:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.691418:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.691433:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.691443:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.700446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.700567:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.700569:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.700571:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.700574:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.700609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.700621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.710031:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.710127:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.710129:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.710132:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.710134:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.710147:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.710172:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.719825:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.719991:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.719993:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.719996:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.719998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.720020:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.720035:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.729328:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.729443:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.729445:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.729448:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.729450:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.729470:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.729480:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.740451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.740592:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.740594:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.740597:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.740599:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.740619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.740629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.749899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.749991:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.749993:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.749996:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.749998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.750011:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.750020:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.760845:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.761000:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.761004:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.761010:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.761015:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.761040:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.761050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.771550:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.771706:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.771708:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.771712:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.771715:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.771737:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.771747:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.781721:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.781831:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.781833:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.781852:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.781854:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.781868:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.781878:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.791392:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.791491:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.791493:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.791496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27d9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.791498:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.791522:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.791533:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.801157:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.801257:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.801259:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.801262:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27da0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.801264:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.801282:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.801307:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.811728:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.811834:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.811836:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.811838:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27da2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.811841:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.811869:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.811879:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.822457:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.822562:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.822564:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.822567:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27da4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.822570:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.822587:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.822597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.831724:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.831839:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.831841:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.831844:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27da6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.831861:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.831875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.831885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.840758:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.840913:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.840915:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.840918:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27da8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.840920:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.840941:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.840951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.851621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.851744:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.851746:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.851750:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27daa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.851752:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.851790:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.851806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.861142:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.861294:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.861296:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.861299:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.861301:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.861320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.861330:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.871956:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.872073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.872075:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.872077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.872080:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.872093:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.872103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.882527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.882635:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.882637:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.882640:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27db0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.882642:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.882658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.882668:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.893419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.893523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.893525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.893528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27db2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.893530:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.893548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.893559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.904366:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.904472:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.904474:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.904477:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27db4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.904480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.904495:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.904505:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.915586:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.915738:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.915739:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.915742:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27db6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.915745:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.915822:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.915834:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.925440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.925535:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.925537:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.925540:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27db8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.925543:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.925561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.925571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.935501:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.935622:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.935624:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.935628:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.935630:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.935644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.935654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.944432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.944551:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.944553:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.944556:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dbc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.944558:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.944577:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.944588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.953996:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.954172:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.954174:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.954177:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dbe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.954180:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.954203:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.954213:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.964945:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.965120:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.965122:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.965125:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dc0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.965127:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.965148:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.965158:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.974235:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.974330:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.974332:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.974335:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dc2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.974337:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.974360:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.974370:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.983876:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.983993:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.983995:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.983998:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dc4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.984000:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.984021:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.984036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773293.995180:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773293.995361:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773293.995363:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773293.995366:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dc6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773293.995368:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773293.995384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773293/real 1521773293]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773305 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773293.995395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.005864:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.005977:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.005979:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.005982:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dc8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.005984:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.005998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.006008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.016777:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.016904:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.016905:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.016909:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.016912:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.016933:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.016943:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.026538:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.026649:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.026650:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.026653:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dcc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.026655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.026669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.026679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.038634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.038746:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.038748:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.038751:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.038753:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.038789:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.038800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.049598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.049716:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.049718:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.049721:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dd0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.049723:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.049744:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.049755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.059526:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.059628:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.059629:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.059632:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dd2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.059634:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.059647:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.059658:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.069258:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.069396:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.069398:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.069400:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dd4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.069403:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.069421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.069432:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.078868:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.079024:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.079026:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.079028:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dd6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.079030:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.079051:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.079061:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.089928:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.090080:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.090082:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.090085:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dd8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.090087:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.090109:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.090119:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.100613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.100739:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.100741:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.100745:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dda sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.100748:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.100761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.100772:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.110176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.110303:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.110304:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.110307:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ddc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.110309:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.110324:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.110334:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.121474:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.121605:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.121607:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.121609:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dde sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.121612:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.121628:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.121660:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.129600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.129788:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.129791:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.129793:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27de0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.129796:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.129834:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.129844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.140636:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.140793:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.140795:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.140798:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27de2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.140801:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.140838:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.140849:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.151454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.151575:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.151577:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.151580:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27de4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.151583:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.151597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.151608:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.161075:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.161202:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.161206:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.161212:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27de6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.161217:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.161251:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.161261:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.171391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.171543:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.171545:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.171548:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27de8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.171550:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.171574:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.171584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.181227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.181343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.181355:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.181359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.181361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.181374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.181384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.191893:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.192023:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.192025:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.192028:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.192030:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.192051:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.192062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.202199:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.202372:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.202374:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.202376:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.202378:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.202400:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.202411:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.211857:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.211980:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.211982:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.211985:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27df0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.211987:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.212008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.212019:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.223026:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.223157:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.223159:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.223162:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27df2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.223164:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.223180:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.223190:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.232383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.232486:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.232488:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.232490:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27df4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.232492:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.232533:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.232544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.241823:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.241993:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.241995:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.241998:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27df6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.242000:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.242022:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.242032:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.252928:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.253053:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.253055:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.253057:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27df8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.253060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.253081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.253097:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.263909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.264021:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.264023:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.264027:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dfa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.264029:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.264043:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.264053:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.274934:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.275125:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.275127:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.275130:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dfc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.275132:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.275154:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.275164:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.284583:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.284693:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.284695:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.284697:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27dfe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.284699:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.284714:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.284725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.294047:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.294155:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.294157:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.294160:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.294162:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.294175:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.294185:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.304986:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.305111:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.305113:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.305116:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.305118:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.305140:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.305150:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.314438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.314628:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.314631:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.314633:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.314636:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.314658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.314669:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.325691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.325864:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.325866:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.325869:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.325871:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.325893:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.325903:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.336982:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.337110:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.337112:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.337115:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.337117:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.337139:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.337149:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.346439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.346545:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.346547:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.346551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.346553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.346572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.346583:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.356142:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.356302:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.356304:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.356307:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.356309:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.356329:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.356339:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.367444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.367564:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.367566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.367569:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.367571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.367589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.367599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.377312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.377452:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.377454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.377456:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.377459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.377480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.377490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.386613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.386731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.386734:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.386736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.386739:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.386760:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.386792:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.396539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.396664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.396666:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.396669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.396671:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.396690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.396700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.407330:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.407489:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.407491:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.407494:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.407496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.407535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.407546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.417677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.417794:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.417796:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.417799:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.417801:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.417823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.417833:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.427824:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.427932:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.427934:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.427938:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.427941:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.427962:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.427972:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.437677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.437837:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.437839:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.437841:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.437844:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.437865:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.437875:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.448728:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.448838:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.448840:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.448843:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.448845:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.448875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.448885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.459759:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.459891:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.459893:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.459896:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.459898:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.459920:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.459930:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.469386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.469520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.469522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.469525:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.469527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.469555:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.469567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.480417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.480551:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.480552:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.480555:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.480558:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.480632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.480644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.491258:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.491364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.491381:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.491384:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.491386:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.491402:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.491411:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.502391:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.502488:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.502490:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.502493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.502495:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.502520:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.502531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.512393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.512513:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.512515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.512520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.512522:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.512553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.512564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.522609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.522770:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.522787:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.522790:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.522792:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.522814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.522830:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.532274:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.532397:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.532399:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.532401:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.532403:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.532424:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.532434:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.541804:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.541920:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.541922:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.541925:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.541927:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.541945:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.541953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.553040:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.553171:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.553173:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.553176:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.553178:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.553199:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.553209:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.562913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.563072:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.563073:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.563076:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.563078:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.563099:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.563110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.572399:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.572499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.572501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.572504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.572512:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.572528:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.572539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.581719:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.581843:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.581845:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.581848:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.581850:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.581872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.581883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.591265:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.591364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.591366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.591386:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.591388:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.591406:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.591416:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.600396:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.600496:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.600498:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.600501:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.600503:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.600537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.600548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.610901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.611087:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.611089:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.611091:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.611094:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.611116:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.611126:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.621691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.621800:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.621802:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.621805:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.621807:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.621846:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.621856:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.631395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.631493:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.631495:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.631498:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.631500:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.631523:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.631534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.641769:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.641943:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.641945:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.641948:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.641950:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.641972:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.641983:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.651237:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.651355:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.651357:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.651359:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.651362:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.651392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.651407:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.662248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.662380:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.662381:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.662384:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.662386:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.662406:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.662416:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.673562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.673699:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.673701:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.673705:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.673708:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.673740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.673753:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.684927:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.685088:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.685090:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.685093:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.685095:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.685116:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.685127:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.695849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.695961:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.695963:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.695965:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.695968:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.695989:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.695999:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.706998:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.707099:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.707101:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.707104:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.707106:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.707119:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.707130:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.716417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.716521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.716523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.716526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.716528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.716544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.716555:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.725925:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.726046:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.726048:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.726051:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.726053:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.726074:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.726084:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.736597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.736711:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.736713:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.736716:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.736718:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.736740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.736750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.747894:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.748000:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.748002:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.748005:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.748007:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.748027:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.748036:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.757620:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.757752:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.757756:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.757764:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e5a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.757769:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.757802:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.757812:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.767382:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.767514:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.767531:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.767534:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.767537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.767561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.767572:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.777831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.777968:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.777970:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.777973:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.777975:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.777996:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.778005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.788814:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.788916:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.788918:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.788921:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.788923:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.788937:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.788953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.800163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.800281:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.800283:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.800286:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e62 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.800288:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.800308:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.800321:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.810200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.810321:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.810323:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.810326:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e64 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.810328:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.810344:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.810363:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.820049:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.820170:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.820172:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.820175:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.820177:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.820197:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.820207:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.830799:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.830931:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.830933:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.830936:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.830938:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.830954:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.830964:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.841830:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.841976:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.841978:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.841983:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.841985:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.842007:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.842017:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.852861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.852987:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.852989:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.852992:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e6c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.852994:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.853008:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.853018:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.862464:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.862565:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.862567:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.862570:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.862572:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.862586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.862597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.872264:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.872367:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.872369:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.872372:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.872374:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.872387:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.872413:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.883455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.883679:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.883681:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.883684:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.883686:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.883709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.883720:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.893439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.893544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.893546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.893549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e74 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.893552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.893569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.893580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.904919:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.905056:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.905058:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.905061:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e76 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.905064:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.905084:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.905094:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.914245:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.914378:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.914380:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.914383:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e78 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.914385:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.914401:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.914411:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.923767:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.923862:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.923864:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.923868:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.923871:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.923902:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.923918:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.934865:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.934976:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.934978:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.934981:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.934983:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.934998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.935008:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.945685:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.945948:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.945950:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.945953:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e7e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.945955:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.945975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.945985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.956831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.956956:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.956958:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.956961:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.956963:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.956983:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.956993:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.966243:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.966335:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.966337:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.966340:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.966343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.966368:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.966394:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.975398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.975541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.975542:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.975545:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.975548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.975580:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.975591:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.986393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.986517:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.986519:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.986522:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.986524:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.986554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.986566:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773294.997718:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773294.997894:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773294.997896:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773294.997899:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773294.997901:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773294.997923:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773294/real 1521773294]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773306 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773294.997934:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.008831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.008949:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.008951:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.008955:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.008958:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.008971:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.008981:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.020414:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.020812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.020814:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.020816:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.020818:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.020856:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.020867:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.031464:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.031592:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.031594:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.031597:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.031600:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.031619:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.031629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.042434:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.042914:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.042916:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.042918:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.042921:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.042958:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.042968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.052589:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.052712:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.052714:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.052717:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.052720:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.052736:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.052746:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.060126:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.060242:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.060244:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.060247:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.060249:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.060270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.060286:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.070803:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.070925:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.070927:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.070930:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.070932:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.070947:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.070957:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.081969:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.082151:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.082153:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.082155:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.082158:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.082180:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.082190:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.091260:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.091362:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.091364:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.091368:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.091371:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.091385:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.091396:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.100931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.101059:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.101060:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.101063:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.101065:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.101087:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.101097:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.111764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.111881:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.111883:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.111886:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27e9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.111888:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.111901:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.111911:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.122870:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.123034:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.123036:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.123039:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ea0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.123041:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.123063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.123073:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.133864:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.133991:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.133993:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.133996:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ea2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.133998:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.134019:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.134029:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.150465:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.150578:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.150580:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.150583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ea4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.150586:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.150600:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.150611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.161535:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.161695:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.161697:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.161700:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ea6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.161702:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.161723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.161733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.172605:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.172739:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.172741:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.172744:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ea8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.172746:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.172762:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.172788:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.182295:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.182417:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.182419:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.182423:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eaa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.182426:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.182448:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.182458:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.192414:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.192503:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.192515:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.192518:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.192520:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.192536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.192547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.208338:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.208533:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.208535:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.208538:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.208540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.208564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.208580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.220046:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.220168:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.220170:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.220172:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eb0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.220175:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.220196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.220206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.229950:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.230067:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.230069:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.230072:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eb2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.230074:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.230095:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.230106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.240951:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.241129:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.241131:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.241134:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eb4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.241136:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.241158:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.241168:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.251874:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.251982:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.251984:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.251987:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eb6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.252005:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.252017:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.252027:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.261363:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.261478:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.261480:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.261482:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eb8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.261485:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.261504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.261539:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.271314:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.271445:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.271447:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.271451:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.271453:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.271474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.271484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.287926:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.288099:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.288101:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.288104:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ebc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.288106:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.288128:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.288138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.297583:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.297688:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.297690:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.297693:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ebe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.297695:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.297716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.297726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.307598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.307724:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.307726:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.307729:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ec0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.307731:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.307753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.307763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.317440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.317565:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.317567:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.317569:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ec2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.317572:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.317591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.317601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.326913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.327013:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.327015:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.327018:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ec4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.327021:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.327049:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.327059:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.336487:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.336603:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.336605:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.336608:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ec6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.336611:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.336624:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.336635:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.353263:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.353391:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.353393:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.353396:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ec8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.353398:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.353425:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.353441:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.364440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.364563:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.364565:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.364572:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.364575:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.364594:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.364604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.375527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.375662:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.375664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.375666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ecc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.375669:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.375690:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.375700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.385093:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.385215:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.385217:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.385220:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ece sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.385222:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.385242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.385252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.394438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.394546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.394548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.394550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ed0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.394553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.394570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.394581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.404338:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.404515:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.404533:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.404535:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ed2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.404538:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.404562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.404572:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.415515:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.415638:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.415640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.415643:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ed4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.415645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.415659:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.415669:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.426528:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.426660:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.426662:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.426665:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ed6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.426667:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.426687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.426697:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.435982:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.436164:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.436166:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.436169:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ed8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.436171:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.436192:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.436203:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.445320:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.445449:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.445451:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.445455:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eda sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.445457:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.445476:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.445487:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.455073:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.455181:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.455183:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.455202:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27edc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.455204:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.455218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.455228:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.466029:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.466162:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.466164:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.466167:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ede sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.466169:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.466190:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.466200:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.476955:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.477113:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.477115:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.477118:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ee0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.477120:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.477142:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.477152:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.488400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.488528:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.488530:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.488532:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ee2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.488535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.488553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.488569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.496652:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.496758:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.496760:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.496763:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ee4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.496765:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.496779:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.496789:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.507775:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.507906:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.507908:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.507910:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ee6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.507913:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.507934:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.507944:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.519380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.519531:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.519533:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.519536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ee8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.519538:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.519605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.519617:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.530451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.530576:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.530579:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.530583:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.530585:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.530601:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.530611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.542023:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.542139:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.542141:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.542144:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.542146:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.542166:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.542176:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.553487:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.553633:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.553635:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.553638:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27eee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.553640:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.553664:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.553674:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.565831:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.566002:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.566004:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.566006:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ef0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.566009:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.566030:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.566040:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.576978:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.577114:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.577116:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.577119:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ef2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.577121:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.577142:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.577152:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.588444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.588614:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.588616:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.588619:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ef4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.588622:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.588645:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.588655:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.600061:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.600223:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.600225:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.600228:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ef6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.600230:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.600252:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.600262:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.611474:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.611632:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.611634:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.611637:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ef8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.611639:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.611662:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.611672:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.623197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.623301:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.623303:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.623307:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27efa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.623309:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.623325:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.623335:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.633158:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.633293:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.633295:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.633298:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27efc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.633300:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.633320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.633335:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.644385:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.644531:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.644533:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.644536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27efe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.644539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.644562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.644573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.654294:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.654465:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.654467:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.654469:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f00 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.654471:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.654491:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.654502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.663901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.664024:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.664026:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.664028:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f02 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.664031:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.664052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.664062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.674949:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.675116:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.675118:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.675120:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f04 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.675123:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.675145:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.675155:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.685049:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.685175:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.685177:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.685180:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f06 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.685182:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.685196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.685206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.695682:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.695826:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.695828:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.695830:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f08 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.695833:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.695854:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.695865:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.705137:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.705241:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.705243:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.705247:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f0a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.705249:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.705264:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.705274:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.714497:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.714635:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.714639:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.714645:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f0c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.714650:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.714671:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.714682:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.725419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.725556:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.725558:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.725561:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f0e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.725563:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.725582:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.725592:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.736602:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.736756:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.736758:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.736761:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f10 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.736764:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.736802:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.736812:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.746402:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.746516:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.746518:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.746521:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f12 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.746523:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.746545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.746556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.755965:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.756068:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.756070:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.756073:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f14 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.756075:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.756093:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.756103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.765311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.765416:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.765418:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.765421:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f16 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.765423:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.765451:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.765467:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.776398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.776515:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.776517:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.776520:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f18 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.776523:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.776545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.776556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.787672:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.787799:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.787801:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.787805:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f1a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.787808:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.787830:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.787840:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.798788:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.798959:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.798961:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.798963:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f1c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.798966:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.798987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.798997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.809953:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.810064:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.810066:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.810068:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f1e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.810071:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.810090:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.810101:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.819777:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.819879:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.819881:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.819884:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f20 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.819901:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.819917:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.819927:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.829393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.829489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.829491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.829494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f22 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.829496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.829521:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.829531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.838729:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.838871:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.838873:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.838876:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f24 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.838878:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.838891:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.838901:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.850079:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.850188:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.850190:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.850193:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f26 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.850195:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.850224:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.850234:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.860952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.861074:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.861076:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.861079:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f28 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.861081:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.861094:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.861104:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.870875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.870998:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.871000:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.871003:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f2a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.871006:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.871019:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.871029:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.880812:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.880979:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.880981:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.880984:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f2c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.880986:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.881007:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.881018:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.890437:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.890545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.890546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.890549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f2e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.890551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.890575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.890585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.900551:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.900659:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.900661:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.900664:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f30 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.900666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.900691:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.900707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.911621:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.911744:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.911746:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.911749:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f32 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.911752:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.911766:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.911776:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.922996:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.923109:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.923111:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.923113:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f34 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.923116:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.923144:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.923154:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.933591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.933757:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.933760:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.933762:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f36 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.933765:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.933803:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.933813:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.944567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.944680:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.944682:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.944685:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f38 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.944688:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.944703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.944713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.954148:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.954266:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.954268:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.954272:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f3a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.954275:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.954311:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.954321:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.965130:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.965296:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.965298:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.965301:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f3c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.965303:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.965324:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.965334:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.976159:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.976326:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.976328:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.976331:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f3e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.976333:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.976362:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.976373:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.987108:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.987233:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.987235:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.987238:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f40 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.987240:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.987261:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.987287:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773295.996572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773295.996710:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773295.996712:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773295.996715:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f42 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773295.996718:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773295.996732:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773295/real 1521773295]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773307 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773295.996743:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.007722:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.007846:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.007848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.007851:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f44 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.007853:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.007875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.007885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.018343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.018467:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.018469:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.018471:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f46 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.018474:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.018494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.018504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.024240:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.025438:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.025440:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.025442:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f48 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.025444:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.025480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.025491:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.036313:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.036480:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.036482:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.036486:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f4a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.036488:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.036553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.036570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.045844:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.046008:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.046010:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.046013:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f4c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.046015:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.046037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.046047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.056557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.056721:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.056723:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.056726:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f4e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.056729:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.056751:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.056761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.067647:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.067817:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.067819:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.067821:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f50 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.067824:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.067845:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.067856:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.077209:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.077341:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.077342:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.077383:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f52 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.077402:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.077419:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.077429:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.088384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.088487:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.088489:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.088492:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f54 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.088494:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.088519:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.088529:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.098135:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.098281:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.098283:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.098286:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f56 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.098288:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.098309:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.098319:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.107645:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.107762:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.107764:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.107766:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f58 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.107769:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.107787:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.107798:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.117616:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.117787:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.117789:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.117792:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f5a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.117795:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.117816:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.117826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.127403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.127521:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.127523:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.127526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f5c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.127528:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.127546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.127557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.136751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.136883:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.136885:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.136887:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f5e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.136889:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.136909:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.136919:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.146395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.146516:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.146518:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.146520:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f60 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.146523:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.146551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.146563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.157418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.157553:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.157555:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.157558:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f62 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.157560:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.157592:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.157604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.168377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.168480:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.168482:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.168485:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f64 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.168487:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.168517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.168536:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.179630:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.179799:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.179801:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.179804:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f66 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.179806:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.179822:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.179832:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.190581:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.190729:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.190731:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.190734:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f68 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.190737:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.190759:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.190785:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.201876:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.202037:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.202039:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.202043:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f6a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.202045:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.202067:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.202077:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.211779:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.211885:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.211887:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.211890:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f6c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.211892:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.211906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.211916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.221552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.221661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.221663:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.221666:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f6e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.221668:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.221682:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.221692:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.231441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.231540:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.231543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.231545:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f70 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.231548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.231562:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.231573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.240861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.240984:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.240986:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.240988:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f72 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.240991:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.241012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.241022:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.249931:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.250046:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.250048:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.250051:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f74 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.250053:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.250066:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.250076:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.261448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.261565:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.261567:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.261570:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f76 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.261573:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.261590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.261600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.271163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.271303:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.271305:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.271307:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f78 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.271310:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.271329:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.271340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.282438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.282594:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.282596:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.282600:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f7a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.282603:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.282627:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.282638:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.293086:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.293197:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.293199:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.293202:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f7c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.293204:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.293218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.293243:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.302451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.302557:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.302559:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.302561:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f7e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.302564:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.302581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.302597:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.312539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.312642:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.312644:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.312647:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f80 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.312649:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.312665:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.312673:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.323753:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.323879:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.323881:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.323884:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f82 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.323886:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.323906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.323916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.333408:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.333518:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.333520:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.333523:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f84 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.333525:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.333545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.333556:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.344574:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.344692:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.344694:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.344697:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f86 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.344700:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.344722:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.344732:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.353846:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.353966:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.353968:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.353971:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f88 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.353973:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.353987:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.353997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.363792:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.363946:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.363948:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.363952:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f8a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.363954:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.363976:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.363986:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.379562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.379705:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.379706:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.379709:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f8c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.379711:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.379733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.379744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.450383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.450532:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.450534:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.450537:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f8e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.450540:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.450564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.450574:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.461237:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.461379:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.461381:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.461384:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f90 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.461386:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.461409:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.461419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.472800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.472923:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.472940:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.472943:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f92 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.472946:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.472967:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.472976:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.484556:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.484708:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.484710:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.484713:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f94 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.484715:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.484737:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.484748:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.494522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.494649:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.494651:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.494654:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f96 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.494656:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.494669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.494679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.556690:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.556846:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.556848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.556850:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f98 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.556853:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.556874:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.556890:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.568181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.568306:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.568308:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.568312:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f9a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.568314:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.568335:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.568354:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.580553:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.580687:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.580690:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.580692:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f9c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.580695:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.580717:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.580727:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.591597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.591726:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.591728:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.591731:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27f9e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.591734:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.591750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.591760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.603371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.603488:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.603490:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.603493:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fa0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.603496:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.603521:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.603531:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.614849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.614989:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.614991:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.614994:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fa2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.614996:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.615012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.615022:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.679698:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.679864:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.679866:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.679868:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fa4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.679871:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.679892:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.679902:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.691453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.691581:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.691583:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.691586:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fa6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.691588:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.691606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.691616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.703649:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.703757:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.703759:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.703762:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fa8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.703764:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.703796:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.703806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.713443:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.713536:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.713539:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.713543:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27faa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.713545:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.713559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.713570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.739300:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.739478:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.739480:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.739483:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.739485:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.739514:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.739525:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.769321:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.769494:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.769496:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.769499:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.769501:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.769548:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.769558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.779688:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.779818:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.779820:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.779822:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fb0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.779825:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.779845:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.779855:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.789441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.789546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.789549:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.789551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fb2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.789554:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.789572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.789588:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.800455:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.800581:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.800583:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.800585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fb4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.800588:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.800605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.800616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.811603:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.811735:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.811738:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.811740:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fb6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.811743:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.811765:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.811775:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.822830:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.822970:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.822972:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.822975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fb8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.822978:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.822991:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.823002:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.833870:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.833993:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.833995:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.833999:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.834002:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.834037:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.834047:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.844866:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.845031:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.845033:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.845036:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fbc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.845039:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.845061:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.845072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.854614:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.854733:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.854735:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.854738:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fbe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.854741:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.854754:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.854765:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.863909:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.864036:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.864038:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.864041:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fc0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.864043:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.864063:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.864074:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.875110:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.875290:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.875292:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.875295:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fc2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.875298:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.875318:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.875328:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.886381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.886495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.886497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.886500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fc4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.886503:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.886530:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.886541:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.897214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.897388:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.897390:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.897393:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fc6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.897396:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.897418:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.897428:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.907304:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.907430:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.907433:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.907435:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fc8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.907438:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.907458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.907469:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.917293:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.917450:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.917454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.917462:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.917467:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.917522:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.917532:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.927389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.927497:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.927499:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.927502:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fcc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.927505:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.927541:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.927557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.938570:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.938701:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.938703:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.938706:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.938708:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.938730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.938741:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.948383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.948532:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.948534:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.948536:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fd0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.948539:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.948569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.948580:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.959441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.959564:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.959566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.959569:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fd2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.959571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.959589:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.959600:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.969529:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.969647:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.969649:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.969652:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fd4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.969655:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.969668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.969679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.981010:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.981174:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.981176:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.981178:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fd6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.981181:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.981203:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.981213:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773296.992136:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773296.992266:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773296.992268:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773296.992271:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fd8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773296.992273:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773296.992293:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773296/real 1521773296]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773308 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773296.992304:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.003417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.003546:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.003548:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.003552:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fda sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.003555:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.003571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.003582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.014728:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.014842:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.014844:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.014847:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fdc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.014849:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.014865:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.014875:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.025545:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.025677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.025679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.025682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fde sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.025684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.025706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.025716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.035548:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.035668:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.035670:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.035673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fe0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.035675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.035695:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.035705:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.045379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.045502:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.045504:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.045512:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fe2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.045514:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.045552:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.045563:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.054812:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.054945:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.054947:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.054950:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fe4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.054952:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.054974:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.054983:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.064542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.064645:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.064647:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.064650:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fe6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.064652:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.064665:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.064680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.075472:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.075594:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.075596:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.075598:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fe8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.075601:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.075614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.075625:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.086452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.086572:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.086574:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.086578:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.086581:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.086596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.086606:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.095748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.095887:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.095889:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.095892:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.095895:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.095916:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.095926:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.106523:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.106637:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.106639:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.106642:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27fee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.106644:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.106658:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.106668:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.117405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.117519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.117521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.117524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ff0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.117526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.117547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.117558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.128400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.128529:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.128530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.128533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ff2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.128535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.128570:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.128582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.139749:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.139860:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.139862:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.139864:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ff4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.139867:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.139888:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.139899:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.151333:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.151457:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.151459:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.151462:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ff6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.151464:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.151485:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.151495:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.162489:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.162623:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.162625:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.162628:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ff8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.162630:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.162646:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.162657:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.173732:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.173938:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.173940:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.173944:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ffa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.173946:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.173968:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.173978:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.181895:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.182013:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.182015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.182017:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ffc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.182020:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.182040:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.182050:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.193317:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.193485:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.193487:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.193490:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x27ffe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.193492:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.193538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.193548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.204595:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.204755:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.204757:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.204760:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28000 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.204762:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.204785:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.204801:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.215596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.215984:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.215986:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.215989:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28002 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.215991:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.216025:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.216035:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.225867:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.226032:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.226034:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.226036:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28004 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.226039:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.226059:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.226069:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.235634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.235753:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.235755:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.235758:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28006 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.235760:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.235796:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.235806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.245214:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.245354:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.245355:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.245358:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28008 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.245360:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.245375:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.245386:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.256415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.256545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.256547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.256551:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2800a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.256553:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.256572:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.256582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.267626:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.267795:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.267797:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.267800:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2800c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.267802:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.267824:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.267834:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.277448:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.277821:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.277823:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.277825:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2800e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.277827:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.277893:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.277904:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.289401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.289522:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.289524:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.289526:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28010 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.289529:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.289559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.289571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.300638:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.300761:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.300763:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.300766:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28012 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.300784:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.300807:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.300817:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.310447:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.310544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.310546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.310549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28014 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.310551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.310565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.310576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.321389:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.321534:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.321536:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.321539:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28016 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.321542:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.321565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.321576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.330848:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.330984:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.330986:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.330989:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28018 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.330991:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.331012:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.331022:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.341703:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.341829:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.341831:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.341835:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2801a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.341838:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.341859:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.341874:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.351404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.351546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.351548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.351551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2801c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.351553:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.351576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.351587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.360998:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.361168:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.361170:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.361172:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2801e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.361175:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.361196:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.361206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.370503:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.370625:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.370627:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.370630:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28020 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.370632:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.370646:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.370656:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.380641:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.380755:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.380758:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.380760:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28022 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.380763:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.380800:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.380810:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.391891:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.392029:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.392031:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.392033:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28024 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.392036:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.392056:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.392066:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.403293:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.403458:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.403460:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.403462:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28026 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.403465:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.403486:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.403496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.414568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.414694:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.414696:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.414699:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28028 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.414701:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.414723:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.414733:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.425764:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.425889:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.425891:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.425895:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2802a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.425897:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.425919:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.425929:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.435622:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.435764:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.435766:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.435769:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2802c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.435771:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.435808:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.435818:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.445187:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.445329:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.445331:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.445334:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2802e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.445336:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.445365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.445375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.455234:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.455559:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.455561:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.455564:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28030 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.455566:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.455591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.455602:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.466442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.466552:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.466554:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.466557:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28032 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.466559:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.466576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.466587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.476270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.476439:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.476441:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.476443:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28034 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.476446:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.476466:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.476482:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.487403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.487527:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.487529:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.487531:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28036 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.487534:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.487556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.487567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.498446:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.498589:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.498591:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.498594:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28038 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.498596:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.498615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.498626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.508055:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.508189:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.508191:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.508195:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2803a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.508197:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.508219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.508229:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.519061:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.519258:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.519260:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.519263:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2803c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.519265:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.519285:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.519296:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.531006:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.531156:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.531158:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.531161:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2803e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.531163:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.531184:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.531195:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.542092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.542247:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.542249:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.542252:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28040 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.542255:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.542275:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.542285:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.554225:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.554381:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.554383:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.554386:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28042 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.554388:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.554411:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.554421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.565469:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.565591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.565593:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.565596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28044 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.565598:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.565614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.565624:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.577287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.577399:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.577401:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.577404:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28046 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.577406:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.577422:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.577432:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.589411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.589531:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.589533:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.589536:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28048 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.589539:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.589561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.589571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.600989:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.601121:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.601123:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.601127:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2804a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.601129:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.601151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.601161:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.613542:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.613656:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.613658:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.613660:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2804c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.613663:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.613685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.613696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.625005:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.625126:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.625128:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.625131:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2804e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.625133:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.625154:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.625169:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.636070:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.636230:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.636232:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.636234:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28050 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.636237:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.636257:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.636268:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.648481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.648608:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.648610:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.648613:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28052 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.648616:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.648631:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.648642:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.659664:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.659811:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.659813:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.659815:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28054 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.659818:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.659840:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.659850:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.671067:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.671213:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.671216:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.671218:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28056 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.671221:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.671242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.671252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.683046:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.683202:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.683204:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.683207:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28058 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.683209:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.683231:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.683241:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.694485:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.694617:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.694619:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.694623:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2805a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.694625:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.694639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.694649:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.706129:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.706248:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.706250:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.706252:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2805c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.706255:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.706275:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.706285:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.716032:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.716212:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.716214:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.716217:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2805e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.716219:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.716240:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.716251:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.725694:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.725805:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.725807:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.725809:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28060 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.725812:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.725832:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.725842:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.735260:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.735440:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.735442:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.735445:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28062 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.735447:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.735467:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.735477:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.746068:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.746200:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.746202:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.746205:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28064 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.746207:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.746222:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.746232:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.756203:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.756364:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.756366:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.756369:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28066 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.756371:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.756394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.756404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.765834:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.765963:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.765965:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.765968:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28068 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.765970:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.765991:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.766006:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.775501:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.775622:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.775624:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.775628:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2806a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.775630:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.775644:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.775654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.786484:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.786608:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.786610:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.786613:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2806c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.786615:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.786629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.786639:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.796261:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.796448:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.796451:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.796453:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2806e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.796455:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.796476:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.796486:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.805649:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.805761:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.805763:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.805766:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28070 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.805768:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.805805:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.805815:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.816470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.816595:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.816597:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.816600:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28072 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.816603:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.816616:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.816627:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.827810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.827939:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.827940:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.827943:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28074 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.827946:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.827967:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.827977:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.839000:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.839190:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.839192:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.839195:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28076 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.839198:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.839219:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.839229:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.848849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.848975:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.848977:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.848979:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28078 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.848982:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.849002:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.849012:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.858275:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.858397:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.858399:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.858403:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2807a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.858406:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.858427:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.858437:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.869407:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.869544:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.869546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.869548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2807c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.869551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.869567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.869578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.878733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.878878:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.878880:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.878882:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2807e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.878885:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.878900:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.878910:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.888983:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.889125:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.889129:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.889135:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28080 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.889140:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.889162:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.889172:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.900479:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.900602:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.900604:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.900607:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28082 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.900609:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.900623:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.900639:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.910252:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.910357:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.910359:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.910362:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28084 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.910365:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.910377:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.910388:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.921060:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.921223:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.921225:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.921227:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28086 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.921230:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.921251:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.921261:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.931093:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.931209:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.931211:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.931214:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28088 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.931216:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.931236:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.931246:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.942502:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.942626:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.942629:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.942633:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2808a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.942635:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.942656:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.942666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.951572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.951677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.951679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.951681:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2808c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.951684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.951698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.951708:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.961547:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.961658:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.961660:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.961663:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2808e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.961666:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.961686:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.961696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.972689:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.972812:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.972829:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.972832:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28090 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.972834:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.972856:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.972866:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.982395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.982492:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.982494:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.982497:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28092 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.982500:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.982525:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.982535:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773297.992311:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773297.992415:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773297.992417:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773297.992435:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28094 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773297.992438:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773297.992455:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773297/real 1521773297]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773309 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773297.992465:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.001826:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.001969:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.001971:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.001974:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28096 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.001976:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.001998:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.002009:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.012941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.013053:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.013055:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.013058:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28098 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.013060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.013081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.013091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.023787:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.023899:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.023901:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.023904:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2809a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.023907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.023927:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.023936:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.033038:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.033160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.033162:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.033165:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2809c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.033167:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.033182:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.033197:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.044045:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.044210:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.044212:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.044214:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2809e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.044217:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.044237:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.044247:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.054411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.054514:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.054516:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.054519:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.054521:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.054544:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.054554:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.065229:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.065356:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.065358:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.065361:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.065363:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.065394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.065404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.074828:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.074976:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.074978:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.074980:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.074983:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.075004:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.075014:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.084451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.084569:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.084571:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.084574:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.084576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.084612:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.084622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.093813:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.093951:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.093952:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.093955:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.093957:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.093977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.093987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.104901:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.105013:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.105015:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.105020:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.105022:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.105042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.105052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.114222:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.114342:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.114344:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.114359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.114361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.114392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.114403:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.125689:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.125847:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.125849:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.125852:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.125854:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.125875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.125885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.132048:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.134455:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.134457:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.134460:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.134462:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.134504:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.134542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.144301:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.144441:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.144443:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.144445:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.144447:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.144468:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.144478:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.155387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.155546:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.155548:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.155551:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.155554:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.155586:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.155598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.165451:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.165593:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.165598:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.165603:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.165608:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.165632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.165647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.176438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.176560:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.176562:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.176565:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.176567:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.176590:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.176601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.186097:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.186209:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.186211:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.186230:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.186232:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.186246:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.186256:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.195874:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.196005:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.196007:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.196010:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.196012:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.196033:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.196043:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.205966:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.206073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.206074:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.206077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.206080:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.206093:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.206103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.215393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.215516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.215518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.215521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.215523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.215542:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.215552:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.226321:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.226452:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.226454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.226456:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.226459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.226479:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.226490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.237895:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.238075:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.238077:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.238080:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.238082:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.238104:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.238115:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.249170:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.249278:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.249281:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.249283:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.249286:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.249299:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.249309:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.260403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.260543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.260545:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.260548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.260550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.260573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.260584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.270797:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.270890:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.270892:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.270896:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.270899:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.270927:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.270937:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.280419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.280558:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.280559:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.280562:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.280564:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.280632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.280644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.289862:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.289967:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.289969:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.289972:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.289974:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.289995:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.290005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.300939:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.301049:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.301051:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.301053:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.301056:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.301077:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.301092:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.310824:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.310936:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.310938:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.310941:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.310943:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.310964:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.310972:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.321881:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.322049:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.322051:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.322054:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.322056:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.322078:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.322088:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.331855:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.331971:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.331973:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.331975:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.331978:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.331991:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.332001:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.343127:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.343282:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.343284:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.343286:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.343289:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.343308:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.343319:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.353225:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.353363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.353365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.353368:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.353371:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.353384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.353395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.364251:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.364432:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.364434:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.364436:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.364439:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.364460:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.364470:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.374210:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.374330:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.374332:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.374334:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.374336:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.374360:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.374371:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.383658:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.383792:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.383794:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.383796:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.383798:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.383811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.383821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.393691:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.393809:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.393811:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.393813:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.393816:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.393844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.393854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.404746:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.404866:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.404868:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.404880:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.404882:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.404898:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.404908:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.415598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.415720:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.415722:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.415725:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.415727:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.415748:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.415758:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.425797:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.425952:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.425954:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.425957:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.425959:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.425981:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.425991:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.436137:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.436277:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.436279:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.436283:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.436285:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.436321:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.436336:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.446554:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.446665:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.446667:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.446669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.446672:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.446685:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.446696:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.458003:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.458128:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.458130:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.458133:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.458135:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.458157:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.458167:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.468160:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.468296:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.468297:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.468300:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.468302:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.468322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.468332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.479263:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.479429:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.479431:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.479434:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.479436:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.479456:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.479467:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.490439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.490564:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.490566:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.490569:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.490571:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.490591:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.490601:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.501662:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.501795:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.501797:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.501800:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.501802:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.501823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.501833:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.511952:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.512084:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.512086:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.512088:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.512091:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.512112:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.512122:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.524850:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.525028:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.525030:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.525034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.525036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.525058:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.525068:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.536384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.536523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.536525:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.536528:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.536531:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.536554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.536565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.548800:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.548926:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.548928:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.548931:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x280fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.548933:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.548955:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.548965:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.560438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.560557:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.560559:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.560562:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28100 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.560564:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.560588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.560598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.572995:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.573127:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.573129:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.573132:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28102 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.573134:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.573156:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.573166:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.590483:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.590597:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.590599:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.590601:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28104 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.590604:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.590617:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.590633:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.602896:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.603083:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.603085:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.603087:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28106 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.603090:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.603111:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.603121:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.614636:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.614766:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.614784:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.614786:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28108 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.614789:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.614811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.614821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.625905:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.626026:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.626028:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.626032:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2810a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.626034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.626056:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.626066:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.637784:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.637899:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.637901:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.637903:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2810c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.637906:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.637927:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.637937:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.648613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.648744:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.648746:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.648749:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2810e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.648752:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.648774:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.648800:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.660272:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.660424:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.660426:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.660429:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28110 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.660431:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.660451:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.660462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.671920:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.672086:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.672088:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.672090:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28112 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.672093:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.672115:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.672125:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.683412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.683528:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.683530:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.683533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28114 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.683535:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.683552:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.683562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.694803:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.694905:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.694907:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.694910:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28116 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.694912:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.694928:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.694953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.705703:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.705837:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.705839:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.705842:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28118 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.705844:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.705865:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.705875:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.717585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.717712:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.717714:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.717719:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2811a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.717721:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.717742:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.717752:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.729310:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.729476:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.729478:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.729481:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2811c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.729484:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.729505:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.729524:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.740462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.740582:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.740584:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.740587:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2811e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.740589:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.740604:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.740619:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.751462:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.751587:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.751589:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.751592:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28120 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.751595:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.751611:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.751621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.762611:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.762770:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.762772:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.762775:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28122 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.762778:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.762811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.762821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.773840:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.773953:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.773955:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.773958:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28124 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.773961:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.773974:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.773984:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.785208:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.785336:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.785338:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.785341:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28126 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.785343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.785368:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.785379:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.795480:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.795603:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.795605:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.795608:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28128 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.795610:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.795627:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.795638:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.805877:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.806051:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.806053:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.806057:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2812a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.806060:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.806081:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.806091:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.817569:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.817684:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.817686:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.817689:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2812c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.817691:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.817705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.817715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.827794:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.827919:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.827921:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.827924:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2812e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.827926:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.827946:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.827955:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.839395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.839534:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.839536:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.839538:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28130 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.839541:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.839564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.839574:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.856021:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.856175:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.856177:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.856180:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28132 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.856182:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.856204:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.856214:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.869606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.869733:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.869735:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.869738:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28134 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.869740:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.869762:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.869788:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.880172:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.880330:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.880332:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.880334:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28136 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.880337:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.880364:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.880375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.891138:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.891261:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.891263:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.891266:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28138 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.891268:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.891288:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.891304:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.902577:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.902737:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.902739:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.902743:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2813a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.902745:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.902783:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.902794:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.913835:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.913958:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.913960:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.913963:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2813c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.913965:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.913979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.913989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.924118:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.924283:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.924285:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.924288:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2813e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.924290:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.924312:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.924322:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.935603:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.935734:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.935737:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.935739:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28140 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.935742:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.935764:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.935790:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.946772:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.946900:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.946902:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.946905:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28142 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.946907:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.946929:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.946939:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.958072:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.958244:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.958246:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.958249:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28144 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.958252:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.958272:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.958283:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.969600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.969731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.969733:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.969736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28146 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.969739:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.969761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.969771:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.980581:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.980709:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.980711:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.980714:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28148 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.980717:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.980739:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.980750:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773298.992447:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773298.992604:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773298.992606:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773298.992610:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2814a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773298.992613:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773298.992633:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773298/real 1521773298]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773310 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773298.992643:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.002215:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.002389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.002391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.002394:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2814c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.002396:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.002420:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.002430:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.013441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.013560:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.013562:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.013565:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2814e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.013567:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.013584:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.013594:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.023618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.023740:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.023742:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.023745:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28150 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.023747:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.023770:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.023796:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.034647:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.034769:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.034771:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.034774:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28152 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.034777:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.034813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.034826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.044572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.044722:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.044724:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.044727:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28154 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.044730:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.044752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.044762:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.054697:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.054834:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.054836:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.054838:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28156 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.054841:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.054862:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.054872:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.066189:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.066361:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.066363:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.066366:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28158 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.066368:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.066391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.066401:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.076452:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.076589:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.076594:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.076601:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2815a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.076607:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.076633:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.076643:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.087613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.087770:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.087772:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.087775:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2815c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.087777:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.087815:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.087825:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.099086:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.099211:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.099213:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.099216:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2815e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.099218:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.099239:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.099249:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.110357:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.110536:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.110538:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.110541:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28160 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.110544:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.110567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.110578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.120276:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.120460:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.120461:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.120464:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28162 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.120466:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.120487:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.120497:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.126394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.127732:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.127734:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.127736:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28164 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.127739:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.127763:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.127775:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.138125:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.138235:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.138237:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.138240:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28166 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.138242:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.138261:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.138271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.148247:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.148361:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.148363:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.148365:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28168 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.148367:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.148386:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.148396:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.159639:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.159761:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.159763:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.159767:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2816a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.159770:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.159791:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.159817:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.170704:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.170805:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.170809:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2816c sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773299.170811:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.170832:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.170847:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.180541:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.180656:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.180658:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.180660:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2816e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.180663:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.180676:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.180686:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.191037:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.191182:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.191184:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.191186:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28170 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.191189:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.191210:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.191220:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.202326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.202480:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.202482:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.202485:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28172 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.202487:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.202532:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.202542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.213478:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.213564:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.213568:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28174 sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773299.213570:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.213584:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.213594:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.223339:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.223460:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.223462:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.223465:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28176 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.223467:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.223487:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.223498:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.233229:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.233343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.233356:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.233359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28178 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.233361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.233374:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.233384:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.244088:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.244220:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.244222:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.244224:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2817a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.244226:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.244292:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.244303:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.253530:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.253648:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.253650:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.253653:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2817c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.253655:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.253668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.253679:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.264637:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.264726:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.264730:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2817e sending 1 waiting 1: failed 5
00000800:00000100:3.0:1521773299.264732:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.264754:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.264764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.276472:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.276594:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.276596:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.276599:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28180 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.276601:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.276615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.276626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.286380:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.286492:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.286494:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.286496:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28182 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.286499:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.286537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.286547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.296693:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.296816:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.296818:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.296821:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28184 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.296823:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.296844:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.296854:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.307958:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.308097:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.308099:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.308102:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28186 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.308104:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.308126:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.308141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.319221:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.319378:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.319380:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.319383:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28188 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.319385:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.319407:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.319418:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.329219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.329332:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.329334:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.329337:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2818a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.329339:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.329365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.329374:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.340482:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.340606:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.340609:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.340612:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2818c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.340614:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.340636:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.340647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.349827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.349967:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.349969:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.349972:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2818e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.349974:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.349995:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.350005:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.359604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.359723:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.359725:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.359728:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28190 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.359730:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.359751:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.359761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.373572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.373967:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.373969:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.373972:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28192 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.373974:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.374047:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.374058:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.383634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.383753:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.383756:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.383758:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28194 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.383761:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.383798:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.383809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.394658:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.394794:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.394796:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.394799:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28196 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.394801:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.394817:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.394827:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.412557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.412715:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.412717:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.412720:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28198 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.412722:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.412744:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.412755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.424237:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.424363:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.424365:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.424368:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2819a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.424370:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.424392:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.424402:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.435405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.435556:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.435576:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.435579:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2819c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.435581:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.435609:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.435620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.445562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.445739:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.445741:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.445744:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2819e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.445747:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.445769:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.445779:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.455498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.455621:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.455623:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.455626:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.455629:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.455643:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.455658:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.465121:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.465239:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.465241:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.465243:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.465246:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.465265:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.465275:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.474550:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.474675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.474677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.474680:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.474682:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.474696:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.474706:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.484417:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.484550:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.484552:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.484554:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.484557:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.484574:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.484585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.494420:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.494527:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.494529:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.494532:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.494534:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.494558:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.494569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.507625:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.507760:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.507762:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.507765:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.507768:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.507799:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.507809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.519246:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.519421:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.519425:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.519428:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.519430:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.519453:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.519464:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.531615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.531748:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.531750:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.531753:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.531755:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.531793:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.531803:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.544294:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.544429:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.544431:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.544434:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.544436:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.544457:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.544468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.556022:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.556229:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.556231:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.556234:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.556236:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.556258:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.556268:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.573553:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.573681:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.573683:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.573686:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.573688:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.573711:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.573721:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.584568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.584697:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.584699:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.584702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.584705:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.584719:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.584729:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.595615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.595755:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.595757:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.595760:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.595762:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.595784:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.595810:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.607866:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.608042:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.608044:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.608047:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.608049:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.608070:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.608085:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.619671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.619807:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.619810:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.619813:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.619815:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.619836:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.619847:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.631561:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.631687:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.631689:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.631692:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.631695:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.631715:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.631725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.642715:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.642887:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.642889:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.642892:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.642894:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.642915:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.642926:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.654554:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.654682:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.654684:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.654687:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.654689:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.654712:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.654722:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.665906:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.666037:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.666039:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.666042:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.666044:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.666065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.666075:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.677789:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.677955:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.677957:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.677960:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.677962:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.677984:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.677994:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.689250:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.689394:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.689396:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.689399:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.689401:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.689420:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.689431:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.707092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.707215:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.707217:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.707220:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.707222:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.707242:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.707252:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.719516:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.719643:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.719646:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.719649:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.719652:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.719667:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.719678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.730558:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.730695:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.730697:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.730700:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.730702:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.730725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.730735:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.742671:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.742800:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.742802:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.742805:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.742807:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.742828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.742838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.757827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.758014:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.758016:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.758019:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.758021:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.758043:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.758053:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.769525:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.769654:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.769656:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.769659:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.769662:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.769684:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.769700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.779502:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.779623:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.779625:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.779628:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.779630:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.779643:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.779654:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.789061:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.789177:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.789179:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.789182:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.789184:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.789205:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.789216:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.799936:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.800103:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.800105:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.800107:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.800110:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.800131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.800141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.817617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.817794:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.817797:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.817800:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.817802:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.817824:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.817834:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.828851:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.828979:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.828981:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.828983:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.828986:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.829007:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.829017:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.838439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.838908:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.838909:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.838912:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.838914:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.838946:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.838957:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.849649:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.849807:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.849809:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.849812:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.849814:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.849836:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.849845:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.860673:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.860800:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.860802:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.860805:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.860807:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.860828:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.860838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.871908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.872042:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.872044:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.872046:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.872049:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.872068:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.872078:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.883318:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.883495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.883497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.883500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.883502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.883551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.883562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.893405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.893518:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.893535:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.893538:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.893540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.893556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.893567:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.912604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.912718:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.912721:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.912724:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.912726:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.912748:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.912759:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.922607:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.922777:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.922779:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.922782:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.922784:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.922807:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.922838:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.934074:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.934217:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.934219:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.934222:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.934224:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.934244:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.934254:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.945223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.945343:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.945356:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.945359:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.945361:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.945382:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.945392:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.955371:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.955541:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.955543:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.955547:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.955549:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.955574:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.955585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.969042:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.969219:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.969221:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.969224:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.969226:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.969247:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.969257:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.980398:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.980530:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.980532:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.980535:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.980537:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.980571:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.980582:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773299.990283:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773299.990426:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773299.990428:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773299.990431:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773299.990433:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773299.990455:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773299.990466:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.000213:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.000375:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.000379:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.000382:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.000384:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.000407:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773299/real 1521773299]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773311 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.000417:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.010223:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.010362:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.010364:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.010366:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x281fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.010369:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.010388:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.010398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.028761:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.028903:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.028905:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.028908:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28200 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.028910:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.028932:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.028942:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.040106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.040285:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.040287:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.040290:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28202 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.040292:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.040313:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.040323:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.051393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.051534:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.051536:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.051539:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28204 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.051541:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.051564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.051575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.062258:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.062374:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.062376:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.062378:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28206 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.062381:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.062396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.062406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.074384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.074542:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.074544:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.074546:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28208 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.074549:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.074614:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.074632:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.085529:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.085683:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.085685:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.085688:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2820a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.085690:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.085704:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.085715:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.096731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.096854:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.096857:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.096860:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2820c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.096862:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.096875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.096885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.107880:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.108007:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.108009:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.108012:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2820e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.108014:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.108036:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.108046:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.117560:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.117717:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.117719:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.117722:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28210 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.117725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.117746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.117757:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.129013:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.129144:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.129146:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.129149:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28212 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.129151:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.129173:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.129183:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.138587:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.138708:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.138710:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.138712:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28214 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.138715:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.138736:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.138747:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.148200:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.148335:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.148337:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.148340:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28216 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.148342:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.148368:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.148378:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.159625:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.159802:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.159804:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.159807:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28218 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.159809:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.159831:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.159841:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.169600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.169724:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.169726:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.169729:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2821a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.169732:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.169752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.169763:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.180706:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.180867:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.180870:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.180873:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2821c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.180875:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.180897:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.180907:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.188271:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.188411:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.188413:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.188416:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2821e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.188418:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.188439:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.188449:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.199598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.199768:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.199770:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.199773:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28220 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.199791:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.199813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.199823:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.210789:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.210930:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.210932:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.210934:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28222 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.210937:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.210958:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.210973:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.222219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.222365:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.222367:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.222370:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28224 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.222372:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.222394:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.222404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.232411:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.232524:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.232526:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.232529:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28226 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.232531:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.232553:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.232564:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.242140:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.242319:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.242321:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.242323:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28228 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.242325:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.242355:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.242365:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.253433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.253569:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.253571:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.253574:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2822a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.253576:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.253597:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.253607:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.263376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.263504:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.263515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.263533:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2822c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.263536:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.263559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.263569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.274429:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.274541:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.274543:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.274546:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2822e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.274548:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.274573:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.274584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.285470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.285592:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.285594:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.285597:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28230 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.285599:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.285615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.285626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.296586:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.296786:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.296788:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.296791:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28232 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.296794:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.296816:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.296826:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.306456:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.306570:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.306572:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.306575:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28234 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.306577:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.306593:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.306604:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.317523:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.317658:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.317660:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.317663:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28236 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.317665:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.317700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.317710:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.328573:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.328738:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.328740:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.328743:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28238 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.328745:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.328783:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.328794:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.338500:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.338629:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.338632:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.338634:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2823a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.338637:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.338651:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.338661:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.348554:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.348673:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.348676:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.348679:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2823c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.348681:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.348702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.348718:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.357867:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.358026:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.358028:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.358031:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2823e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.358033:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.358053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.358063:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.369300:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.369421:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.369423:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.369426:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28240 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.369428:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.369450:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.369460:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.379023:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.379147:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.379149:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.379152:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28242 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.379154:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.379175:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.379183:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.389925:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.390075:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.390077:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.390079:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28244 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.390082:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.390101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.390111:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.399427:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.399562:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.399564:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.399566:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28246 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.399569:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.399632:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.399644:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.409403:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.409515:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.409532:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.409535:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28248 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.409537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.409567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.409578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.418889:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.419023:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.419025:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.419028:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2824a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.419030:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.419052:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.419062:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.435552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.435693:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.435696:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.435699:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2824c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.435701:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.435728:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.435739:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.446297:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.446499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.446501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.446504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2824e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.446530:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.446554:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.446565:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.452393:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.452517:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.452519:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.452522:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28250 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.452524:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.452538:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.452549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.467628:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.467766:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.467768:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.467771:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28252 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.467774:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.467788:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.467813:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.479443:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.479588:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.479590:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.479593:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28254 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.479596:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.479618:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.479629:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.489219:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.489373:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.489375:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.489378:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28256 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.489381:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.489396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.489412:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.499192:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.499320:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.499322:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.499324:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28258 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.499327:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.499355:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.499365:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.516615:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.516765:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.516767:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.516786:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2825a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.516788:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.516811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.516821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.528218:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.528396:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.528401:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.528403:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2825c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.528406:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.528428:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.528439:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.540263:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.540406:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.540408:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.540411:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2825e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.540413:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.540435:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.540445:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.552479:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.552635:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.552637:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.552640:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28260 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.552642:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.552659:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.552669:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.564248:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.564426:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.564428:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.564430:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28262 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.564433:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.564456:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.564466:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.576281:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.576515:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.576517:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.576520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28264 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.576522:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.576547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.576558:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.587561:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.587693:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.587695:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.587698:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28266 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.587700:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.587720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.587731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.600339:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.600535:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.600538:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.600540:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28268 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.600543:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.600566:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.600577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.612975:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.613103:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.613105:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.613108:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2826a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.613110:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.613131:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.613141:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.624563:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.624747:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.624750:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.624753:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2826c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.624755:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.624793:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.624803:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.635951:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.636079:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.636081:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.636084:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2826e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.636087:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.636151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.636161:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.647736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.647885:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.647887:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.647889:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28270 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.647892:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.647913:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.647928:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.659024:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.659169:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.659171:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.659174:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28272 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.659176:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.659197:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.659207:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.671326:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.671475:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.671477:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.671480:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28274 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.671483:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.671498:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.671515:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.682733:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.683000:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.683002:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.683005:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28276 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.683007:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.683026:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.683037:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.694049:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.694173:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.694175:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.694177:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28278 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.694180:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.694200:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.694210:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.705412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.705542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.705544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.705547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2827a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.705549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.705575:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.705585:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.716960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.717147:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.717150:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.717153:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2827c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.717155:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.717178:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.717188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.728479:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.728608:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.728610:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.728613:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2827e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.728615:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.728631:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.728641:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.740435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.740555:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.740557:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.740560:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28280 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.740562:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.740579:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.740590:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.752386:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.752500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.752502:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.752514:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28282 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.752517:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.752539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.752550:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.764444:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.764593:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.764595:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.764598:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28284 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.764600:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.764666:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.764678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.775491:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.775615:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.775618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.775620:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28286 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.775623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.775637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.775647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.784796:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.784921:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.784922:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.784925:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28288 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.784927:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.784949:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.784959:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.794324:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.794453:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.794455:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.794458:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2828a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.794461:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.794475:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.794491:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.805338:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.805466:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.805469:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.805472:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2828c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.805474:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.805493:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.805503:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.814861:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.815000:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.815002:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.815005:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2828e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.815007:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.815028:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.815038:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.825941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.826064:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.826066:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.826069:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28290 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.826071:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.826092:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.826102:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.837196:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.837971:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.837972:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.837975:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28292 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.837977:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.838059:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.838070:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.848789:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.848932:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.848934:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.848937:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28294 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.848939:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.848961:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.848971:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.858559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.858684:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.858686:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.858689:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28296 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.858692:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.858708:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.858718:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.868527:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.868650:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.868652:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.868655:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28298 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.868657:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.868678:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.868688:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.879556:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.879694:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.879696:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.879699:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2829a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.879701:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.879721:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.879731:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.890537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.890661:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.890664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.890667:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2829c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.890669:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.890683:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.890694:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.901604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.901791:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.901793:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.901796:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2829e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.901798:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.901814:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.901824:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.911381:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.911731:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.911733:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.911736:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.911739:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.911761:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.911787:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.922590:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.922767:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.922769:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.922787:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.922789:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.922811:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.922821:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.933776:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.933936:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.933938:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.933940:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.933943:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.933964:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.933979:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.943765:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.943885:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.943887:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.943890:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.943892:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.943912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.943922:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.953227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.953355:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.953357:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.953360:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.953362:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.953384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.953395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.963742:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.963884:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.963886:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.963889:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.963891:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.963912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.963922:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.974657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.975147:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.975150:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.975153:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.975155:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.975191:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.975201:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.985220:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.985389:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.985391:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.985394:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.985396:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.985419:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.985429:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773300.995327:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773300.995453:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773300.995455:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773300.995458:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773300.995460:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773300.995481:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773300/real 1521773300]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773312 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773300.995491:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.006305:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.006478:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.006480:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.006483:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.006485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.006505:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.006538:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.016273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.016426:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.016428:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.016431:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.016433:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.016455:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.016466:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.026578:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.026693:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.026695:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.026698:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.026700:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.026716:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.026726:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.036851:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.037029:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.037031:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.037034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.037037:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.037058:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.037069:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.047263:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.047377:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.047379:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.047382:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.047385:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.047415:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.047425:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.058528:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.058658:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.058661:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.058664:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.058666:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.058687:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.058698:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.069418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.069532:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.069534:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.069537:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.069540:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.069557:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.069573:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.080498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.080619:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.080621:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.080624:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.080626:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.080642:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.080652:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.090596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.090754:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.090756:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.090759:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.090762:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.090799:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.090809:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.100384:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.100497:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.100499:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.100502:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.100504:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.100551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.100561:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.109968:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.110073:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.110075:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.110077:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.110080:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.110093:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.110103:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.119537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.119677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.119679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.119682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.119684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.119702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.119712:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.129312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.129444:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.129446:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.129449:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.129451:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.129466:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.129476:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.140652:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.140793:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.140796:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.140799:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.140801:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.140823:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.140833:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.150170:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.150303:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.150305:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.150308:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.150310:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.150330:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.150340:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.160090:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.160246:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.160248:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.160251:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.160253:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.160273:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.160283:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.169941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.170068:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.170070:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.170073:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.170075:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.170098:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.170108:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.182181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.182374:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.182376:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.182379:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.182381:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.182395:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.182406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.191819:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.191965:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.191967:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.191969:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.191972:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.191993:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.192003:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.201439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.201570:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.201572:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.201574:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.201577:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.201596:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.201612:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.210880:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.210996:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.210998:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.211001:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.211003:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.211016:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.211026:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.220440:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.220543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.220546:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.220549:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.220551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.220565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.220576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.230240:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.230378:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.230380:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.230382:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.230385:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.230405:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.230415:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.239570:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.239693:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.239696:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.239698:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.239701:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.239720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.239730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.250552:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.250685:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.250687:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.250690:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.250693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.250715:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.250725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.259936:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.260067:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.260069:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.260072:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.260075:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.260096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.260106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.270933:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.271058:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.271060:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.271063:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.271065:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.271086:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.271096:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.282332:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.282500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.282502:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.282505:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.282513:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.282551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.282562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.293752:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.293872:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.293874:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.293877:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.293879:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.293899:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.293909:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.304944:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.305087:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.305090:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.305092:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.305095:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.305116:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.305126:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.316343:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.316533:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.316535:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.316538:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.316540:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.316564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.316574:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.326769:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.326916:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.326918:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.326921:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282f0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.326923:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.326945:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.326955:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.337786:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.337917:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.337919:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.337922:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282f2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.337924:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.337945:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.337961:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.349181:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.349320:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.349322:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.349324:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282f4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.349327:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.349356:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.349366:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.359141:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.359321:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.359323:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.359326:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282f6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.359328:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.359357:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.359368:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.370323:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.370442:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.370444:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.370447:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282f8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.370449:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.370469:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.370479:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.380661:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.380819:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.380821:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.380823:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282fa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.380826:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.380853:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.380863:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.391908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.392040:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.392043:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.392046:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282fc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.392048:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.392069:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.392079:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.402418:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.402549:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.402551:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.402554:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x282fe sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.402556:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.402583:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.402593:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.411892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.412042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.412044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.412046:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28300 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.412049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.412062:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.412072:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.423134:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.423249:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.423251:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.423254:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28302 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.423256:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.423275:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.423284:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.433239:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.433366:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.433368:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.433370:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28304 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.433373:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.433410:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.433421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.444735:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.444941:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.444943:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.444945:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28306 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.444948:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.444986:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.444997:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.457163:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.457333:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.457335:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.457337:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28308 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.457340:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.457384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.457395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.468609:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.468740:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.468742:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.468745:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2830a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.468747:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.468769:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.468779:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.480481:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.480615:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.480618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.480621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2830c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.480623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.480639:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.480655:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.492379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.492502:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.492504:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.492515:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2830e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.492518:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.492534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.492544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.504141:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.504303:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.504305:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.504308:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28310 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.504310:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.504357:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.504367:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.515454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.515598:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.515600:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.515603:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28312 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.515605:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.515623:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.515634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.526818:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.527029:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.527031:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.527034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28314 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.527036:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.527074:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.527084:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.538272:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.538438:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.538440:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.538443:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28316 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.538445:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.538466:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.538476:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.549549:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.549676:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.549678:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.549681:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28318 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.549683:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.549705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.549716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.561875:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.562045:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.562048:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.562050:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2831a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.562053:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.562091:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.562101:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.574102:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.574255:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.574258:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.574261:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2831c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.574263:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.574300:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.574310:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.585993:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.586133:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.586135:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.586138:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2831e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.586140:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.586179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.586189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.597409:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.597545:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.597547:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.597550:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28320 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.597552:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.597576:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.597587:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.609376:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.609516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.609518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.609521:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28322 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.609523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.609547:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.609557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.621324:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.621457:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.621459:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.621462:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28324 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.621464:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.621485:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.621496:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.632592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.632717:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.632719:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.632722:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28326 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.632724:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.632740:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.632756:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.643971:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.644137:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.644139:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.644142:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28328 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.644144:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.644183:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.644193:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.654944:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.655092:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.655094:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.655096:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2832a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.655099:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.655136:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.655147:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.665904:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.666043:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.666046:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.666049:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2832c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.666051:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.666088:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.666099:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.677273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.677438:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.677440:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.677443:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2832e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.677446:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.677466:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.677477:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.688606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.688726:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.688728:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.688731:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28330 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.688734:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.688750:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.688760:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.700908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.701070:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.701072:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.701075:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28332 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.701077:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.701117:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.701128:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.712422:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.712531:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.712533:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.712536:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28334 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.712538:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.712561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.712572:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.724015:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.724187:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.724189:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.724192:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28336 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.724194:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.724232:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.724243:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.735879:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.736004:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.736006:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.736008:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28338 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.736011:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.736042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.736052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.747056:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.747178:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.747180:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.747182:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2833a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.747185:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.747217:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.747227:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.758791:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.758985:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.758988:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.758991:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2833c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.758993:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.759032:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.759042:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.770092:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.770217:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.770219:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.770222:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2833e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.770224:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.770262:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.770273:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.780907:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.781039:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.781041:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.781044:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28340 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.781046:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.781084:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.781101:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.792239:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.792444:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.792446:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.792449:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28342 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.792451:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.792475:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.792486:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.803453:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.803577:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.803579:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.803581:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28344 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.803584:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.803606:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.803616:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.815881:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.816019:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.816021:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.816023:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28346 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.816026:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.816065:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.816075:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.827205:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.827359:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.827361:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.827364:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28348 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.827367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.827388:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.827398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.838738:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.838914:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.838916:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.838919:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2834a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.838921:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.838943:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.838953:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.848449:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.848567:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.848570:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.848573:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2834c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.848575:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.848592:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.848603:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.859504:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.859643:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.859645:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.859648:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2834e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.859650:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.859670:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.859680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.869862:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.870013:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.870015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.870018:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28350 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.870021:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.870042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.870052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.880637:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.880808:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.880810:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.880813:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28352 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.880815:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.880837:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.880847:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.890568:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.890691:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.890693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.890696:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28354 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.890698:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.890720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.890730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.900505:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.900643:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.900645:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.900648:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28356 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.900650:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.900663:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.900673:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.909946:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.910069:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.910071:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.910074:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28358 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.910076:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.910097:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.910107:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.920731:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.920907:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.920909:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.920912:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2835a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.920914:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.920936:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.920951:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.930419:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.930523:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.930526:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.930529:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2835c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.930532:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.930550:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.930560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.940373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.940499:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.940501:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.940504:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2835e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.940515:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.940537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.940548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.950377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.950504:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.950513:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.950516:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28360 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.950533:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.950556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.950566:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.961681:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.961861:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.961863:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.961866:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28362 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.961868:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.961890:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.961900:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.971454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.971586:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.971588:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.971591:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28364 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.971593:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.971612:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.971622:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.982394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.982527:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.982529:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.982531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28366 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.982534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.982551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.982562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773301.992176:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773301.992295:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773301.992297:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773301.992300:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28368 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773301.992302:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773301.992322:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773301/real 1521773301]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773313 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773301.992332:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.002205:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.002367:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.002369:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.002371:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2836a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.002374:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.002396:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.002406:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.011737:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.011836:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.011839:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.011842:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2836c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.011844:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.011873:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.011883:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.021696:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.021844:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.021847:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.021849:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2836e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.021851:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.021872:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.021882:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.031435:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.031542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.031544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.031547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28370 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.031549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.031567:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.031577:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.041633:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.041835:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.041837:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.041839:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28372 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.041842:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.041863:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.041873:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.051769:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.051910:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.051912:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.051915:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28374 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.051917:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.051939:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.051954:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.061811:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.061937:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.061939:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.061941:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28376 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.061944:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.061956:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.061966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.072135:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.072256:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.072258:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.072261:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28378 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.072263:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.072283:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.072293:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.081773:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.081945:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.081947:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.081950:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2837a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.081952:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.081975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.081985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.088305:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.088423:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.088427:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.088429:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2837c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.088432:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.088451:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.088462:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.098259:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.098387:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.098389:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.098392:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2837e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.098394:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.098416:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.098427:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.109265:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.109377:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.109379:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.109382:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28380 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.109384:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.109405:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.109414:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.120499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.120669:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.120671:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.120673:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28382 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.120675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.120692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.120702:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.130487:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.130608:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.130610:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.130613:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28384 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.130615:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.130629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.130639:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.136497:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.138202:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.138203:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.138206:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28386 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.138208:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.138262:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.138273:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.149275:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.149434:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.149436:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.149439:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28388 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.149442:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.149465:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.149476:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.160498:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.160660:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.160664:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.160670:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2838a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.160675:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.160702:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.160713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.170657:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.170845:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.170848:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.170851:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2838c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.170853:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.170875:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.170885:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.180884:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.181013:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.181015:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.181017:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2838e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.181019:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.181041:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.181056:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.191218:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.191337:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.191339:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.191342:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28390 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.191344:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.191370:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.191380:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.201497:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.201615:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.201617:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.201620:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28392 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.201622:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.201637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.201647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.211374:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.211516:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.211533:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.211536:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28394 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.211538:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.211561:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.211571:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.221322:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.221452:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.221454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.221457:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28396 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.221459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.221474:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.221485:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.231559:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.231705:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.231707:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.231710:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28398 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.231713:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.231733:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.231744:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.243771:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.243949:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.243951:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.243953:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2839a sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.243956:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.243977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.243987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.254886:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.255026:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.255029:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.255032:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2839c sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.255034:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.255054:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.255064:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.264903:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.265009:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.265011:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.265014:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2839e sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.265016:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.265043:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.265053:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.275369:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.275504:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.275515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.275534:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283a0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.275537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.275559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.275569:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.286543:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.286737:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.286739:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.286742:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283a2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.286744:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.286766:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.286777:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.298028:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.298164:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.298166:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.298169:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283a4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.298171:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.298191:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.298201:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.309712:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.309914:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.309916:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.309919:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283a6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.309921:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.309944:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.309954:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.321427:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.321566:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.321567:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.321570:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283a8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.321573:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.321611:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.321628:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.331387:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.331513:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.331515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.331518:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283aa sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.331537:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.331566:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.331578:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.342557:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.342689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.342693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.342696:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ac sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.342698:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.342720:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.342730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.354606:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.354754:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.354756:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.354759:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ae sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.354761:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.354782:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.354793:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.365165:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.365280:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.365282:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.365285:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283b0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.365288:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.365307:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.365316:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.376979:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.377099:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.377101:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.377104:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283b2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.377106:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.377127:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.377138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.387598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.387722:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.387724:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.387727:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283b4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.387729:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.387745:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.387755:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.398929:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.399114:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.399116:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.399119:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283b6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.399121:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.399143:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.399153:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.410395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.410495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.410497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.410500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283b8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.410502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.410534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.410544:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.421383:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.421495:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.421497:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.421500:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ba sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.421502:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.421588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.421599:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.431412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.431519:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.431522:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.431525:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283bc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.431528:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.431545:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.431555:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.441373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.441487:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.441489:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.441492:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283be sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.441494:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.441534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.441545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.451755:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.451910:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.451912:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.451914:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283c0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.451917:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.451938:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.451948:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.462937:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.463073:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.463075:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.463077:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283c2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.463080:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.463101:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.463117:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.474052:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.474160:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.474162:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.474165:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283c4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.474167:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.474181:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.474206:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.485099:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.485227:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.485229:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.485232:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283c6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.485234:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.485255:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.485265:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.496224:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.496379:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.496381:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.496383:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283c8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.496386:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.496408:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.496419:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.507485:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.507639:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.507641:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.507644:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ca sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.507647:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.507669:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.507680:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.519929:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.520098:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.520101:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.520104:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283cc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.520106:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.520128:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.520138:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.532406:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.532571:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.532573:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.532576:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ce sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.532578:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.532602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.532613:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.544629:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.544753:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.544755:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.544758:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283d0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.544760:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.544780:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.544806:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.557012:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.557148:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.557150:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.557152:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283d2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.557155:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.557218:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.557229:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.569112:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.569253:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.569255:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.569258:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283d4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.569260:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.569281:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.569291:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.581277:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.581591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.581593:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.581596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283d6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.581598:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.581624:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.581634:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.593540:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.593662:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.593664:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.593667:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283d8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.593669:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.593689:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.593700:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.605585:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.605750:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.605752:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.605755:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283da sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.605757:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.605794:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.605804:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.617836:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.617962:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.617965:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.617968:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283dc sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.617970:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.617992:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.618007:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.629253:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.629416:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.629418:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.629421:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283de sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.629423:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.629444:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.629454:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.642080:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.642190:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.642192:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.642206:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283e0 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.642208:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.642224:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.642234:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.653995:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.654174:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.654177:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.654179:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283e2 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.654182:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.654203:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.654214:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.667291:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.667452:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.667454:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.667457:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283e4 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.667459:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.667480:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.667490:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.679161:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.679336:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.679338:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.679340:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283e6 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.679343:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.679369:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.679379:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.690913:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.691047:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.691049:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.691052:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283e8 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.691054:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.691075:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.691086:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.703227:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.703388:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.703390:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.703393:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ea sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.703395:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.703417:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.703428:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.714400:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.714514:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.714518:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.714520:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ec sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.714523:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.714546:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.714557:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.725441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.725598:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.725600:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0a280) failed: 5
00000800:00000100:3.0:1521773302.725603:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ee sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.725605:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.725629:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.725639:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.735236:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.735362:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.735364:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.735367:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.735369:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.735391:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.735402:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.744604:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.744725:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.744727:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.744730:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.744732:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.744754:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.744764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.755469:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.755583:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.755585:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.755588:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.755590:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.755605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.755615:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.765415:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.765528:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.765530:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.765533:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.765535:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.765560:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.765576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.775385:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.775486:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.775488:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.775491:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.775493:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.775517:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.775528:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.786571:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.786696:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.786698:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.786701:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.786703:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.786725:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.786736:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.796558:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.796696:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.796700:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.796702:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.796704:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.796730:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.796741:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.807848:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.808018:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.808020:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.808023:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x283ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.808025:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.808048:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.808058:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.818433:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.818542:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.818544:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.818547:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28401 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.818549:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.818569:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.818579:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.829441:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.829563:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.829564:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.829567:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28403 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.829569:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.829588:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.829598:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.840461:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.840580:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.840582:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.840585:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28405 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.840588:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.840610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.840621:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.851978:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.852110:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.852112:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.852114:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28407 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.852117:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.852139:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.852149:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.862139:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.862288:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.862290:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.862293:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28409 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.862295:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.862316:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.862326:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.873094:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.873223:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.873226:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.873228:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2840b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.873231:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.873245:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.873255:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.884395:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.884529:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.884533:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.884535:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2840d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.884538:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.884605:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.884617:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.894197:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.894331:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.894333:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.894336:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2840f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.894338:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.894365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.894375:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.905423:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.905521:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.905523:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.905526:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28411 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.905528:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.905556:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.905572:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.918724:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.918929:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.918931:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.918934:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28413 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.918936:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.918958:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.918968:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.928479:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.928591:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.928593:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.928596:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28415 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.928599:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.928612:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.928623:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.939655:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.939774:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.939776:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.939779:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28417 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.939782:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.939813:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.939822:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.949270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.949382:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.949384:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.949387:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28419 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.949389:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.949411:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.949421:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.964713:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.964882:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.964884:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.964887:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2841b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.964889:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.964912:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.964922:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.974613:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.974727:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.974730:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.974733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2841d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.974736:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.974758:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.974768:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.984315:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.984455:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.984457:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.984459:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2841f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.984462:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.984482:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.984492:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773302.994271:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773302.994401:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773302.994403:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773302.994405:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28421 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773302.994408:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773302.994429:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773302/real 1521773302]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773314 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773302.994439:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.004165:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.004280:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.004282:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.004285:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28423 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.004287:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.004306:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.004315:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.017342:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.017486:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.017488:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.017491:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28425 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.017493:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.017537:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.017548:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.027634:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.027788:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.027791:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.027793:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28427 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.027796:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.027834:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.027844:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.038892:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.039069:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.039071:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.039073:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28429 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.039076:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.039097:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.039107:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.048940:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.049066:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.049068:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.049071:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2842b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.049073:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.049094:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.049110:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.058826:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.058953:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.058958:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.058960:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2842d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.058962:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.058975:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.058985:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.072108:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.072240:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.072242:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.072245:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2842f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.072247:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.072270:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.072280:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.083459:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.083604:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.083606:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.083609:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28431 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.083611:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.083637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.083647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.093608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.094041:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.094043:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.094045:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28433 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.094047:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.094121:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.094132:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.104827:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.104948:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.104951:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.104953:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28435 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.104956:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.104977:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.104987:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.114537:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.114686:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.114688:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.114690:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28437 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.114692:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.114706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.114716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.125499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.125676:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.125678:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.125681:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28439 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.125684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.125706:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.125716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.132108:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.133304:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.133306:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.133308:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2843b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.133311:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.133376:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.133387:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.144562:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.144689:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.144692:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.144695:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2843d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.144697:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.144719:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.144730:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.154216:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.154332:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.154334:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.154337:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2843f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.154339:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.154365:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.154376:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.164281:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.164399:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.164400:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.164403:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28441 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.164405:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.164427:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.164437:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.181533:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.181684:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.181686:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.181689:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28443 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.181691:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.181714:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.181725:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.191442:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.191543:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.191545:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.191548:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28445 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.191551:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.191565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.191581:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.201394:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.201633:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.201635:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.201637:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28447 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.201640:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.201666:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.201676:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.212470:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.212580:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.212582:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.212585:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28449 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.212587:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.212602:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.212611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.223736:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.223873:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.223875:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.223878:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2844b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.223880:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.223894:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.223904:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.234880:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.235005:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.235008:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.235011:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2844d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.235013:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.235032:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.235042:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.244597:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.244733:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.244735:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.244738:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2844f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.244740:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.244760:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.244771:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.253916:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.254042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.254044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.254046:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28451 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.254049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.254069:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.254079:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.263432:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.263538:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.263540:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.263543:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28453 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.263545:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.263559:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.263570:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.275600:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.275728:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.275730:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.275733:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28455 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.275735:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.275757:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.275768:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.284608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.284725:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.284727:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.284730:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28457 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.284732:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.284752:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.284761:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.296379:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.296500:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.296503:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.296514:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28459 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.296517:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.296539:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.296549:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.314662:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.314795:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.314797:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.314800:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2845b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.314803:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.314825:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.314835:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.324993:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.325186:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.325189:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.325192:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2845d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.325195:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.325216:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.325227:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.334438:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.334542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.334544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.334547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2845f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.334550:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.334568:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.334584:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.345635:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.345766:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.345768:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.345771:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28461 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.345773:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.345810:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.345820:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.355596:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.355724:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.355726:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.355729:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28463 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.355731:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.355753:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.355764:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.368608:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.368757:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.368759:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.368762:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28465 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.368764:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.368786:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.368795:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.378546:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.378677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.378679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.378682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28467 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.378684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.378698:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.378709:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.387889:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.388014:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.388016:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.388018:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28469 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.388021:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.388042:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.388052:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.398912:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.399184:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.399186:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.399189:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2846b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.399192:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.399213:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.399223:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.413499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.413637:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.413640:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.413642:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2846d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.413645:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.413667:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.413677:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.423522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.423664:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.423666:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.423669:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2846f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.423672:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.423692:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.423703:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.433819:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.433951:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.433953:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.433956:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28471 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.433958:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.433979:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.433989:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.445185:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.445356:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.445358:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.445361:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28473 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.445363:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.445384:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.445395:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.454992:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.455126:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.455128:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.455131:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28475 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.455133:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.455154:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.455165:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.465210:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.465338:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.465340:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.465343:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28477 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.465349:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.465371:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.465381:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.476479:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.476616:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.476618:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.476621:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28479 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.476623:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.476637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.476653:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.487592:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.487736:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.487737:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.487740:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2847b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.487743:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.487777:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.487803:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.497617:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.497775:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.497794:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.497797:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2847d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.497799:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.497821:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.497831:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.509008:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.509143:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.509145:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.509148:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2847f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.509150:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.509172:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.509182:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.520454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.520584:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.520587:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.520589:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28481 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.520592:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.520610:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.520620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.532802:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.532937:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.532939:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.532941:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28483 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.532944:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.532965:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.532975:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.544378:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.544520:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.544522:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.544524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28485 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.544527:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.544549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.544560:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.556618:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.556745:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.556747:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.556749:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28487 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.556752:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.556789:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.556799:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.567748:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.567879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.567881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.567884:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28489 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.567886:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.567908:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.567917:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.579759:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.579953:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.579955:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.579958:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2848b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.579960:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.579982:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.579992:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.592405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.592526:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.592529:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.592531:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2848d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.592534:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.592564:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.592576:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.605849:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.606030:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.606032:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.606034:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2848f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.606037:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.606058:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.606068:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.617217:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.617393:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.617395:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.617398:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28491 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.617400:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.617423:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.617433:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.628970:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.629124:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.629126:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.629129:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28493 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.629131:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.629153:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.629168:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.640677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.640865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.640866:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.640869:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28495 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.640872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.640893:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.640903:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.652234:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.652377:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.652379:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.652382:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28497 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.652384:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.652406:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.652416:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.663677:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.663810:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.663812:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.663814:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28499 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.663817:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.663837:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.663848:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.674917:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.675042:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.675044:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.675047:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2849b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.675049:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.675070:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.675080:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.687076:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.687270:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.687273:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.687276:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2849d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.687278:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.687298:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.687308:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.698706:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.698862:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.698864:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.698867:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2849f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.698869:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.698890:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.698900:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.709810:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.709928:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.709930:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.709933:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284a1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.709935:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.709955:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.709966:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.722430:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.722542:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.722544:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.722547:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284a3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.722549:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.722565:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.722575:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.733721:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.733891:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.733893:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.733896:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284a5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.733898:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.733920:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.733931:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.745630:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.745790:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.745792:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.745810:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284a7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.745812:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.745835:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.745845:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.757751:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.757879:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.757881:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.757884:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284a9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.757886:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.757906:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.757916:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.769220:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.769363:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.769365:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.769368:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284ab sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.769370:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.769404:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.769415:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.779209:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.779359:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.779362:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.779365:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284ad sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.779367:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.779382:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.779398:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.790312:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.790461:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.790463:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.790465:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284af sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.790468:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.790488:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.790499:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.800404:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.800534:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.800536:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.800539:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284b1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.800541:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.800608:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.800620:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.809803:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.809933:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.809935:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.809938:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284b3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.809940:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.809954:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.809963:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.819069:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.819186:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.819188:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.819190:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284b5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.819193:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.819214:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.819224:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.830412:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.830519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.830521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.830523:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284b7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.830526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.830549:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.830559:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.840439:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.840556:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.840558:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.840561:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284b9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.840563:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.840581:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.840592:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.850410:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.850519:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.850521:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.850524:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284bb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.850526:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.850543:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.850553:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.859994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.860118:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.860121:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.860124:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284bd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.860126:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.860147:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.860157:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.871467:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.871591:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.871593:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.871596:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284bf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.871598:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.871617:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.871628:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.882487:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.882625:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.882627:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.882630:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284c1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.882632:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.882655:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.882666:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.893538:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.893674:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.893676:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.893678:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284c3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.893681:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.893703:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.893713:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.903287:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.903465:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.903467:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.903470:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284c5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.903472:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.903492:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.903502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.913302:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.913439:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.913441:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.913444:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284c7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.913446:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.913468:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.913484:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.922994:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.923118:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.923122:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.923128:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284c9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.923133:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.923179:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.923189:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.932938:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.933125:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.933128:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.933130:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284cb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.933133:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.933155:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.933165:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.942536:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.942656:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.942659:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.942662:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284cd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.942664:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.942678:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.942689:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.953785:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.953924:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.953926:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.953929:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284cf sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.953931:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.953951:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.953961:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.963426:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.963575:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.963577:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.963580:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284d1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.963583:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.963646:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.963658:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.974362:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.974486:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.974488:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.974491:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284d3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.974493:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.974536:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.974547:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.985369:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.985489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.985491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.985494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284d5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.985496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.985534:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.985545:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773303.996533:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773303.996663:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773303.996666:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773303.996668:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284d7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773303.996671:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773303.996693:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773303/real 1521773303]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773315 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773303.996704:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.006377:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.006489:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.006491:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.006494:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284d9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.006496:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.006535:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.006546:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.023941:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.024050:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.024052:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.024055:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284db sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.024057:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.024073:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.024083:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.034089:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.034211:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.034214:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.034216:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284dd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.034219:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.034240:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.034251:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.045331:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.045475:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.045477:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.045480:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284df sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.045482:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.045531:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.045542:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.054775:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.054908:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.054910:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.054913:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284e1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.054915:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.054934:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.054950:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.064499:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.064617:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.064619:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.064621:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284e3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.064624:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.064637:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.064647:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.074273:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.074393:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.074395:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.074398:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284e5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.074400:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.074421:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.074432:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.084990:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.085093:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.085095:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.085098:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284e7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.085100:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.085117:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.085127:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.095999:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.096192:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.096194:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.096197:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284e9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.096199:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.096220:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.096231:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.105598:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.105717:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.105720:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.105722:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284eb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.105725:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.105746:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.105757:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.116908:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.117106:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.117108:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.117111:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284ed sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.117114:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.117136:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.117146:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.125106:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.125222:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.125224:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.125227:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284ef sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.125229:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.125249:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.125259:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.134570:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.134675:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.134677:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.134680:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284f1 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.134682:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.134697:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.134707:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.145522:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.145638:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.145640:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.145643:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284f3 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.145646:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.145668:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.145678:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.156880:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.157037:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.157039:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.157042:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284f5 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.157044:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.157064:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.157073:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.166949:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.167068:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.167070:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.167073:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284f7 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.167075:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.167096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.167106:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.178270:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.178405:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.178407:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.178409:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284f9 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.178412:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.178426:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.178436:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.189148:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.189277:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.189279:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.189282:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284fb sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.189284:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.189320:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.189336:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.199976:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.200136:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.200139:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.200142:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284fd sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.200144:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.200166:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.200176:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.209331:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.209450:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.209452:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.209454:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x284ff sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.209456:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.209492:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.209502:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.219958:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.220053:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.220055:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.220057:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28501 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.220060:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.220088:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.220098:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.231289:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.231428:0:1676:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.231430:0:1676:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.231433:0:1676:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28503 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.231435:0:1676:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.231458:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.231468:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.242373:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.242473:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.242475:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.242477:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28505 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.242480:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.242494:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.242504:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.253551:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.253677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.253679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.253682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28507 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.253684:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.253705:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.253716:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.263022:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.263149:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.263151:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.263153:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28509 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.263156:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.263177:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.263188:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.272960:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.273076:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.273078:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.273081:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2850b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.273083:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.273096:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.273105:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.284004:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.284114:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.284118:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.284121:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2850d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.284123:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.284151:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.284161:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.293572:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.293708:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.293710:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.293712:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2850f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.293715:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.293805:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.293817:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.304564:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.304677:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.304679:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.304682:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28511 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.304685:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.304700:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.304711:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.313899:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.314025:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.314027:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.314030:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28513 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.314032:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.314053:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.314063:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.324534:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.324684:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.324685:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.324688:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28515 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.324690:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.324756:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.324789:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.334591:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.334691:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.334693:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.334696:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28517 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.334698:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.334713:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.334723:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.345454:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.345576:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.345578:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.345581:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28519 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.345583:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.345601:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.345611:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.355275:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.355371:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.355373:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.355376:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2851b sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.355378:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.355393:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.355404:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.364401:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.365188:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.365192:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.365194:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2851d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.365196:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.365260:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.365271:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.376468:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.376585:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.376587:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.376590:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2851f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.376592:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.376615:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.376626:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.387567:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.387686:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.387688:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.387691:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28521 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.387693:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.387709:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.387719:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.398539:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.398696:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.398697:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.398700:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28523 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.398702:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.398767:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.398795:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.408367:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.408478:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.408480:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.408482:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28525 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.408485:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.408498:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.408534:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.419709:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.419865:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.419867:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d09428) failed: 5
00000800:00000100:3.0:1521773304.419870:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x28527 sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.419872:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:1.0:1521773304.419893:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:1.0:1521773304.419903:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.431884:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.431979:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.431981:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0aca0) failed: 5
00000800:00000100:3.0:1521773304.431984:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2858d sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.431986:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773304.432006:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773304.432015:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
00000100:02000000:0.0:1521773304.443405:0:1734:0:(import.c:1542:ptlrpc_import_recovery_state_machine()) oak-OST0013-osc-ffff8800b8462000: Connection restored to 10.0.2.102@o2ib5 (at 10.0.2.102@o2ib5)
00000800:00000100:3.0:1521773304.443513:0:1675:0:(o2iblnd_cb.c:3470:kiblnd_complete()) FastReg failed: 6
00000800:00000100:3.0:1521773304.443515:0:1675:0:(o2iblnd_cb.c:3481:kiblnd_complete()) RDMA (tx: ffffc90001d0aca0) failed: 5
00000800:00000100:3.0:1521773304.443518:0:1675:0:(o2iblnd_cb.c:967:kiblnd_tx_complete()) Tx -> 10.0.2.102@o2ib5 cookie 0x2858f sending 1 waiting 0: failed 5
00000800:00000100:3.0:1521773304.443520:0:1675:0:(o2iblnd_cb.c:1914:kiblnd_close_conn_locked()) Closing conn to 10.0.2.102@o2ib5: error -5(waiting)
00000100:00000400:0.0:1521773304.443551:0:1737:0:(client.c:2114:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1521773304/real 1521773304]  req@ffff88035478fc00 x1595673331398768/t0(0) o4->oak-OST0013-osc-ffff8800b8462000@10.0.2.102@o2ib5:6/4 lens 4688/1464 e 0 to 1 dl 1521773316 ref 2 fl Rpc:eX/2/ffffffff rc 0/-1
00000100:02000400:0.0:1521773304.443562:0:1737:0:(import.c:178:ptlrpc_set_import_discon()) oak-OST0013-osc-ffff8800b8462000: Connection to oak-OST0013 (at 10.0.2.102@o2ib5) was lost; in progress operations using this service will wait for recovery to complete
Debug log: 26537 lines, 26537 kept, 0 dropped, 0 bad.